Tag Archives: Unix

What is the fastest Linux or Unix flavour?

What is the fastest Linux or Unix flavour?
Everyone talks about Linux and Unix in generic terms. Has anyone ever done any metrics to find which performs best or is the fastest for their applications? How would the newer Ubuntu compare to something like Red Hat? Where does CentOS or Debian come into play? How does Solaris versus Open Solaris measure up? We are trying to decide which flavour is best for our application. It is both written in Java and C++.

Can anyone answer this?

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Could F# be the secret future for which Microsoft will start branching into the Linux, Unix, and open source worlds?

Could F# be the secret future for which Microsoft will start branching into the Linux, Unix, and open source worlds?

As some quant developers secretly know, they hold the key to future world of technology. Let’s face it; they are the ones who have kept C++ alive and well. They need the speed as this language is the fastest processor on the planet. No debate here, end of discussion on that. Quant developers have kept old legacy databases like Sybase relevant. Don’t ask me why?
As we know, the latest 2010 edition of Visual Studio from Microsoft included a relatively unknown programming language called F#. Could it hold the future to Microsoft’s first foray into Linux or Unix? While installing the Linux Mono project (thanks to Novell for that one), F# enables you to go up against any dynamic programming language like Python or Ruby or even R.
I have never used it, but I can historically say, most languages (outside of Java) die a fairly quick non relevancy within the enterprise world unless it has the backing of Microsoft. Well this one does since it owned and developed by Microsoft.
When you look at the history of C#, it was meant to be the modern Java killer. It did evolve from J#. Today, it takes eight years for C# to become more popular that Visual Basic. Hooray for them on that!
Back to F#, I have heard from some influential sources that F# is gaining some traction as a language to go with. In terms of modern and establish large investment banks, F# is gaining some momentum on new projects. As a result, does this mean F# could be the next mainstream programming language Microsoft may push over the next ten years. Or will it die a slow death like J# or other dynamic programming languages Microsoft tried to push.
When you look at Microsoft’s attitude towards the open source community, I get very confused on their strategies with open source interpretations of their Iron Python and Iron Ruby projects. When you look at F#, it will become mainstream otherwise it would never have seen light of day in their modern Visual Studio 2010 suite.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

How do I choose Microsoft .Net technologies like C++ or C# versus Red Hat Linux or Solaris Unix platforms for my Java or C++ quant development?

How do I choose Microsoft .Net technologies like C++ or C# versus Red Hat Linux or Solaris Unix platforms for my Java or C++ quant development?
As I mentioned in a previous post as being part of these influential Linked In groups, there was a very interesting one which was the most popular operating system and C++ compilers out there in the world of quant. I have already discussed my views of Linux versus Microsoft Windows. For those that don’t know, I would go with Linux based version if you need latency to near zero. Windows technology might not cut it.
As part of the C++ compiler question, why was there a resounding popularity for Windows based compilers like Visual Studio and Microsoft related programming languages like C++ or C#? I mean, once you support these languages, it is safe to say you will become a fully Microsoft shop. As I have consulted with numerous investment banks and even researched a bunch, it seems larger banks like Morgan Stanley, Citi, or whoever use a hybrid approach. It sounds like many of CIti’s derivative trading platforms are deployed on Microsoft Windows servers while other departments like their Risk maybe deployed on a Linux or Unix based platform like Solaris. I mean it gets kind of confusing for those starting out in Quant development.
I think one suggestion for those go into this career is to choose a better cross platform language like Java or C++. Also, if you go with a C++ technology, ensure the frameworks (or libraries) you choose are cross platform as well. This is one reason why I like Java backed web frameworks like Google Web Toolkit as they are cross platform and browser independent. You obviously do not get that with Microsoft .Net based technologies which of course include C ++ or C#. As I said, choose well based on these technologies and programming languages for your early quant development days.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Is Red Hat the next enterprise level Linux flavour to take off instead of Oracle or Sun Solaris Unix version?

Is Red Hat the next enterprise level Linux flavour to take off instead of Oracle or Sun Solaris Unix version?
I am part of a few Linked In groups that seems to suggest who is winning in the world of quant development. These seem to be high influential Linked In groups that were started by probably one of the quant recruiters on the planet, Dominic O’Connor from the United Kingdom.
There is always these discussions of the open source world versus what Microsoft stands which of course includes the .Net world. As I see it, over the last few years I come from more of the open source with solid movements of free Linux versions like Ubuntu, CentOS, Debian, and the like. I am unaware of any serious bank, bulge brackets, or hedge funds using these Linux flavours. From my interviews with big organizations like Bloomberg, Morgan Stanley, and Citi, they all seem to be in bed with Solaris. There is always talk of migrating their systems to Red Hat. The only large organization that I know that is in bed with Linux is Goldman Sachs. After working with a guy from Canada Pension Plan Investment Board, he confirmed that Goldman is very proprietary but a lot of their early technology was based off of Red Hat. They also were the early of adopters of Red Hat and legitimized them before they went commercial with their Linux version. I can also confirm that Pension Plan Investment Board (CPP-IB) is using Red Hat for the all their production systems outside of their Windows based systems. So there is another one.
Why aren’t any of the other mentioned companies swiftly moving over to Red Hat? From what I gather, it seems they are hesitant in rushing their large code based legacy systems to another platform from their Solaris platforms. As it stands as Oracle owns all this Sun technology like Sun, many are very scared on what they will do with it. So here they sit waiting for what happens to Red hat. As it stands they are another alternative to an enterprise level Linux/Unix based platform to choose from.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

This is easily the best and simplest C or C++ Posix compliant Linux Unix multi thread pthread tutorial I have found

This is easily the best and simplest C or C++ Posix compliant Linux Unix multi thread pthread tutorial I have found

http://www.mobydisk.com/softdev/techinfo/pthreads_tutorial/

POSIX threads tutorial
This is a short tutorial on how to code POSIX threads in C++ on GNU/Linux. For up-to-date information on the Linux implementation of pthreads called LinuxThreads, check out Xavier Leroy’s page.
What you need
Any modern Linux should be able to do this. For specific requirements, check outthe link above. This tutorial does not explain what threads are, how they work, or how to synchronize multiple threads. It assumes you know what you plan to use threads for, and that you have a good reference on synchronizing threads. If you don’t have any of this information, then visit the Links section at the bottom of this page.
Compiling this demo
At your Linux prompt type:
g++ threaddemo.cpp -o threaddemo -lpthread -D_REENTRANT
then:
./threaddemo
Program description
This sample program will create 2 threads. One thread displays the word “ping” and the other displays “pong.” The resulting output will look something like this:
ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,
ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,
ping,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,
pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,
pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,
pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,pong,ping,
Done!
Sample source
The sample source code is threaddemo.cpp. You will also need Morgan McGuire’s great kbhit.cpp which makes things a little easier.
The how-to
The pthreads library hides out in the pthread.h file, as you can see below.
#include #include
#include “kbhit.cpp”

// On Linux, you must compile with the -D_REENTRANT option. This tells
// the C/C++ libraries that the functions must be thread-safe
#ifndef _REENTRANT
#error ACK! You need to compile with _REENTRANT defined since this uses threads
#endif
Whenever compiling an application for pthreads, you must use the -D_REENTRANT option to inform the C/C++ libraries that they are in multithread mode. The #ifndef _REENTRANT line verifies that this is the case before allowing the compile to continue. Further, you should only make calls to other libraries that were compiled with -D_REENTRANT. If you need to know if your favorite library is compiled with this code, seek the author or the documentation. For more information about this subject see the links at the bottom of this page.
volatile int bStop = false;
The bStop variable is used to tell the threads when to stop. If the variable is not declared volatile, then the compiler may think that the value is always false, and may not even pay attention to it! More later.
////////////////////////////////////////////////////////////////////////////////
// Starting point for a thread
////////////////////////////////////////////////////////////////////////////////
void * runMe(void *generic_pointer)
{
// Do this until we are told to stop
while (!bStop)
{
// Output some text
cout << static_cast(generic_pointer) << ","; // Relenquish the CPU for another thread pthread_yield(); } return NULL; } This routine will be executed in each of the 2 threads that we will create. The function takes a void pointer, and returns a void pointer. This is necessary, since it is a requirement of the pthread_create function that we are about to use. The parameter, and the return value are arbitrary; that is, they can be used for whatever purpose you wish. In this demo, the parameter is assumed to be a char * that holds a string for the thread to display. In a real application, this parameter may be a pointer to a structure with information about what the thread needs to do. For example, imagine a multithreaded game, with a thread controlling each of the enemy characters. This parameter may be a pointer to the character's information, with the character position, intelligence, weapons, etc. The return value is not used here. It could be treated similarly to the parameter, in that it can be used to return some arbitrary data to the parent thread. It could be a pointer to an integer return value indicating success or failure, or an exception structure, or error message. The basic function of this thread is to display a message over and over, while not using up the entire CPU. The thread avoids hogging the CPU by calling pthread_yield() after displaying the message. This function tells the OS that the thread isn't busy and something else can go happen now. Note that the OS may not wait until the thread calls pthread_yield to switch threads. It may happen at any point. This call just tells the OS that this would be a good time to do so. More on this later. The thread will stop displaying the message once the value of bStop becomes true. At this point, the thread will exit with a return value of NULL. This return statement does not return control to the caller: it stops the thread from executing entirely. This is the equivalent to calling pthread_exit(NULL). pthread_t nThreadID1, nThreadID2; Every thread has an ID. Nothing special there. pthread_create(&nThreadID1, NULL, runMe, szMessage1); pthread_create(&nThreadID2, NULL, runMe, szMessage2); This code is kinda like doing: runMe(szMessage1); runMe(szMessage2); Except that the code executes in the background, instead of sequentially. After this code, there are 3 threads: the parent, and 2 children. Let's check out the pthread_create call. The first parameter is the address of nThreadID. pthread_create will stuff the thread ID into this. The second parameter has all sorts of nifty stuff about the thread: priority, scheduling, stack size, whatever. We don't care about any of this so it is NULL. Next is the name of the function to use as the starting point for the thread. The function must take a void * and return a void *. Lots of tutorials do this incorrectly, and must cast the function with some crazy C-style weirdness. If you declare the function exactly right, you can just place the name there and nothing else. The last function is the generic parameter, which we use for the message. while (!_kbhit()) { pthread_yield(); } At this point the child threads are running. The parent thread now loops until the user presses a key. It is crucial that this loop call pthread_yield() or the child threads may not execute. This is because the parent thread would spend all its time checking for a key, and never giving anyone else a chance to work. This behavior of threads varies with operating system, kernel, processor, phase of the moon, etc. Pthreads makes no guarantee that each thread executes "fairly." Threads may be interrupted at any time, for any length of time. If this isn't appropriate, then you may need some thread synchronization code, or to provide a hint during the pthread_create call about how you want the threads to behave. bStop = true; This changes the value of the bStop variable. When the threads check this value they will exit. If this value were not declared volatile, then it is possible that the threads may not notice it, and continue running forever! pthread_join(nThreadID1, NULL); pthread_join(nThreadID2, NULL); The pthread_join call tells a thread to wait until the specified thread has exited. This is why we saved the thread IDs. What happens if we remove this code? In this demo, nothing. The child threads will stop anyway, once the parent thread exits. In more complex code, you may need to wait for your threads to free their memory, close open files, etc. before your application exits. The pthread_join call also allows you to see the return value of the thread. Remember the void * return value? The second parameter to pthread_join() tells you where to put this value. For this application, we don't care what they returned. Further improvements A nifty improvement (left to the reader, of course) would be to have each thread return the number of times it displayed the message. In a perfect world, it would be the same for each thread. But as we know, threads aren't guaranteed fair excecution time! In theory, this code is POSIX compliant (except for the kbhit.cpp) and should run on other operating systems. You may wish to compile this code and run it on Coolix or Niftyux or FreeBVD or whatever and see if it works. Links Most of these are pretty much the first things you find in a search for threads, linux, and POSIX or some combination thereof. • Xavier Leroy is the author of LinuxThreads and his page has lotsa info. • posix threads programming • DMOZ links on Linux threads programming

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Know both Linux or Unix top command and ps command to clear Bloomberg interview question

Know both Linux or Unix top command and ps command to clear Bloomberg interview question

This is for questions on how to diagnose for memory leaks or run away processes

top
Process viewer, find the CPU-intensive programs currently running. See ps for explanations of the field descriptors.
Syntax
top options

Options
-b Run in batch mode; don’t accept command-line input.
Useful for sending output to another command or to a file.
-c Show command line in display instead of just command name.
-d delay
Specify delay between refreshes.
-i Suppress display of idle and zombie processes.
-n num
Update display num times, then exit.
-p pid
Monitor only processes with the specified process ID.
-q Refresh without any delay.
If user is privileged, run with highest priority.
-s Secure mode. Disable some (dangerous) interactive commands.
-S Cumulative mode. Print total CPU time of each process,
including dead child processes.

Interactive commands

space Update display immediately.
A Sort by age, with newest first.
c Toggle display of command name or full command line.
f, F Add fields to display or remove fields from the display.
h, ? Display help about commands and the status of secure and cumulative modes.
k Prompt for process ID to kill and signal to send (default is 15) to kill it.
i Toggle suppression of idle and zombie processes.
l Toggle display of load average and uptime information.
^L Redraw screen.
m Toggle display of memory information.
M Sort tasks by resident memory usage.
n, # Prompt for number of processes to show.
If 0 is entered, show as many as will fit on the screen
(default).
N Sort numerically by process ID.
o, O Change order of displayed fields.
P Sort tasks by CPU usage (default).
q Exit.
r Apply renice to a process. Prompt for PID and renice value. Suppressed in secure mode.
s Change delay between refreshes.
Prompt for new delay time, which should be in seconds.
(Suppressed in secure mode.)
S Toggle cumulative mode. (See the -S option.)
t Toggle display of processes and CPU states information.
T Sort tasks by time/cumulative time.
W Write current setup to ~/.toprc. This is the recommended way to write a top configuration file.
Top displays per-process CPU usage (not total server load) it is useful for seeing how much work the machine is doing now compared to some point in the past.
At the top of the display output there are three numbers representing the number of processes waiting for the CPU now, an average for the past five minutes, and an average for the past fifteen minutes. These three numbers are the “load average”
Top should only be used to compare the load average on two different machines if they have an identical configuration (both hardware and software.)
ps
Process status, information about processes running in memory. If you want a repetitive update of this status, use top.
Syntax

ps option(s)
ps [-L]

Options
-L List all the keyword options
This version of ps accepts 3 kinds of option:

-Unix98 options may be grouped and must be preceeded by a dash.
BSD options may be grouped and must not be used with a dash.
–GNU long options are preceeded by two dashes.

Options of different types may be freely mixed. The PS_PERSONALITY environment variable provides more detailed control of ps behavior.

The Options below are listed side-by-side (unless there are differences).
Simple Process Selection:
-A a select all processes (including those of other users)
-a select all with a tty except session leaders
-d select all, but omit session leaders
-e select all processes
g really all, even group leaders (does nothing w/o SunOS settings)
-N negate selection
r restrict output to running processes
T select all processes on this terminal
x select processes without controlling ttys
–deselect negate selection

Process Selection by List:

-C select by command name
-G select by RGID (supports names)
-g select by session leader OR by group name
–Group select by real group name or ID
–group select by effective group name or ID
-p p –pid select by process ID (PID)
-s –sid select by session ID
-t –tty select by terminal (tty)
-u U select by effective user ID (supports names)
-U select by RUID (supports names)
–User select by real user name or ID
–user select by effective user name or ID

-123 implied –sid
123 implied –pid

Output Format Control:

-c Different scheduler info for -l option
-f Full listing
-j j Jobs format
-l l Long format
-O O Add the information associated with the space or comma separated
list of keywords specified, after the process ID, in the default
information display.

-o o Display information associated with the space or comma separated
list of keywords specified.
–format user-defined format
s display signal format
u display user-oriented format
v display virtual memory format
X old Linux i386 register format
-y do not show flags; show rss in place of addr

Output Modifiers:
C use raw CPU time for %CPU instead of decaying average
c true command name
e show environment after the command
f ASCII-art process hierarchy (forest)
-H show process hierarchy (forest)
h do not print header lines (repeat header lines in BSD personality)
-m m show all threads
-n set namelist file
n numeric output for WCHAN and USER
N specify namelist file
O sorting order (overloaded)
S include some dead child process data (as a sum with the parent)
-w w wide output
–cols set screen width
–columns set screen width
–forest ASCII art process tree
–html HTML escaped output
–headers repeat header lines
–no-headers print no header line at all
–lines set screen height
–nul unjustified output with NULs
–null unjustified output with NULs
–rows set screen height
–sort specify sorting order
–width set screen width
–zero unjustified output with NULs

Information:
-V V print version
L list all format specifiers
–help print help message
–info print debugging info
–version print version

Obsolete:
A increase the argument space (DecUnix)
M use alternate core (try -n or N instead)
W get swap info from … not /dev/drum (try -n or N instead)
k use /vmcore as c-dumpfile (try -n or N instead)
NOTES
The “-g” option can select by session leader OR by group name. Selection by session leader is specified by many standards, but selection by group is the logical behavior that several other operating systems use. This ps will select by session leader when the list is completely numeric (as sessions are). Group ID numbers will work only when some group names are also specified.

The “m” option should not be used. Use “-m” or “-o” with a list. (“m” displays memory info, shows threads, or sorts by memory use)

The “h” option varies between BSD personality and Linux usage (not printing the header) Regardless of the current personality, you can use the long options –headers and –no-headers

Terminals (ttys, or screens of text output) can be specified in several forms: /dev/ttyS1, ttyS1, S1. Obsolete “ps t” (your own terminal) and “ps t?” (processes without a terminal) syntax is supported, but modern options (“T”,”-t” with list, “x”, “t” with list) should be used instead.

The BSD “O” option can act like “-O” (user-defined output format with some common fields predefined) or can be used to specify sort order. Heuristics are used to determine the behavior of this option. To ensure that the desired behavior is obtained, specify the other option (sorting or formatting) in some other way.

For sorting, BSD “O” option syntax is O[+|-]k1[,[+|-]k2[,…]] Order the process listing according to the multilevel sort specified by the sequence of short keys from SORT KEYS, k1, k2, … The `+’ is quite optional, merely re-iterating the default direction on a key. `-‘ reverses direction only on the key it precedes.
The O option must be the last option in a single command argument, but specifications in successive arguments are catenated.

GNU sorting syntax is –sortX[+|-]key[,[+|-]key[,…]]
Choose a multi-letter key from the SORT KEYS section. X may be any convenient separator character. To be GNU-ish use `=’. The `+’ is really optional since default direction is increasing numerical or lexicographic order. For example, ps jax –sort=uid,-ppid,+pid

This ps works by reading the virtual files in /proc. This ps does not need to be suid kmem or have any privileges to run. Do not give this ps any special permissions.

This ps needs access to a namelist file for proper WCHAN display. The namelist file must match the current Linux kernel exactly for correct output.

To produce the WCHAN field, ps needs to read the System.map file created when the kernel is compiled. The search path is:
$PS_SYSTEM_MAP
/boot/System.map-`uname -r`
/boot/System.map
/lib/modules/`uname -r`/System.map
/usr/src/linux/System.map
/System.map
The member used_math of task_struct is not shown, since crt0.s checks to see if math is present. This causes the math flag to be set for all processes, and so it is Programs swapped out to disk will be shown without command line arguments, and unless the c option is given, in brackets.

%CPU shows the cputime/realtime percentage. It will not add up to 100% unless you are lucky. It is time used divided by the time the process has been running.

The SIZE and RSS fields don’t count the page tables and the task_struct of a proc; this is at least 12k of memory that is always resident. SIZE is the virtual size of the proc (code+data+stack).

Processes marked are dead processes (so-called”zombies”) that remain because their parent has not destroyed them properly. These processes will be destroyed by init(8) if the parent process exits.
PROCESS FLAGS
ALIGNWARN 001 print alignment warning msgs
STARTING 002 being created
EXITING 004 getting shut down
PTRACED 010 set if ptrace (0) has been called
TRACESYS 020 tracing system calls
FORKNOEXEC 040 forked but didn’t exec
SUPERPRIV 100 used super-user privileges
DUMPCORE 200 dumped core
SIGNALED 400 killed by a signal

PROCESS STATE CODES
D uninterruptible sleep (usually IO)
R runnable (on run queue)
S sleeping
T traced or stopped
Z a defunct (“zombie”) process

For BSD formats and when the “stat” keyword is used,
additional letters may be displayed:
W has no resident pages
< high-priority process N low-priority task L has pages locked into memory (for real-time and custom IO) ENVIRONMENT VARIABLES and PERSONALITY (posix,linux,bsd,sun) Examples: List every process on the system using standard syntax: ps -e List every process on the system using BSD syntax: ps ax List the top 10 CPU users. ps -e -o pcpu -o pid -o user -o args | sort -k 1 | tail -21r List every process except those running as root (real & effective ID) ps -U root -u root -N List every process with a user-defined format: ps -eo pid,tt,user,fname,tmout,f,wchan Odd display with AIX field descriptors: ps -o "%u : %U : %p : %a" Print only the process IDs of syslogd: ps -C syslogd -o pid= When displaying multiple fields, part of the output may be truncated, to avoid this supply a width to the arguments: ps -e -o user:20,args. Since ps cannot run faster than the system and is run as any other scheduled process, the informa

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

The point here is to understand the difference between a Linux or Unix pipe and a named pipe

The point here is to understand the difference between a Linux or Unix pipe and a named pipe

Introduction to Named Pipes
Sep 01, 1997 By Andy Vaught
in
• SysAdmin
A very useful Linux feature is named pipes which enable different processes to communicate.
One of the fundamental features that makes Linux and other Unices useful is the “pipe”. Pipes allow separate processes to communicate without having been designed explicitly to work together. This allows tools quite narrow in their function to be combined in complex ways.
A simple example of using a pipe is the command:
ls | grep x
When bash examines the command line, it finds the vertical bar character | that separates the two commands. Bash and other shells run both commands, connecting the output of the first to the input of the second. The ls program produces a list of files in the current directory, while the grep program reads the output of ls and prints only those lines containing the letter x.
The above, familiar to most Unix users, is an example of an “unnamed pipe”. The pipe exists only inside the kernel and cannot be accessed by processes that created it, in this case, the bash shell. For those who don’t already know, a parent process is the first process started by a program that in turn creates separate child processes that execute the program.
The other sort of pipe is a “named” pipe, which is sometimes called a FIFO. FIFO stands for “First In, First Out” and refers to the property that the order of bytes going in is the same coming out. The “name” of a named pipe is actually a file name within the file system. Pipes are shown by ls as any other file with a couple of differences:
% ls -l fifo1
prw-r–r– 1 andy users 0 Jan 22 23:11 fifo1|
The p in the leftmost column indicates that fifo1 is a pipe. The rest of the permission bits control who can read or write to the pipe just like a regular file. On systems with a modern ls, the | character at the end of the file name is another clue, and on Linux systems with the color option enabled, fifo| is printed in red by default.
On older Linux systems, named pipes are created by the mknod program, usually located in the /etc directory. On more modern systems, mkfifo is a standard utility. The mkfifo program takes one or more file names as arguments for this task and creates pipes with those names. For example, to create a named pipe with the namepipe1 give the command:
mkfifo pipe
The simplest way to show how named pipes work is with an example. Suppose we’ve created pipe as shown above. In one virtual console1, type:
ls -l > pipe1
and in another type:
cat < pipe Voila! The output of the command run on the first console shows up on the second console. Note that the order in which you run the commands doesn't matter. If you haven't used virtual consoles before, see the article “Keyboards, Consoles and VT Cruising” by John M. Fisk in the November 1996 Linux Journal. If you watch closely, you'll notice that the first command you run appears to hang. This happens because the other end of the pipe is not yet connected, and so the kernel suspends the first process until the second process opens the pipe. In Unix jargon, the process is said to be “blocked”, since it is waiting for something to happen. One very useful application of named pipes is to allow totally unrelated programs to communicate with each other. For example, a program that services requests of some sort (print files, access a database) could open the pipe for reading. Then, another process could make a request by opening the pipe and writing a command. That is, the “server” can perform a task on behalf of the “client”. Blocking can also happen if the client isn't writing, or the server isn't reading. Pipe Madness Create two named pipes, pipe1 and pipe2. Run the commands: echo -n x | cat - pipe1 > pipe2 &
cat pipe1
On screen, it will not appear that anything is happening, but if you run top (a command similar to ps for showing process status), you’ll see that both cat programs are running like crazy copying the letter x back and forth in an endless loop.
After you press ctrl-C to get out of the loop, you may receive the message “broken pipe”. This error occurs when a process writing to a pipe when the process reading the pipe closes its end. Since the reader is gone, the data has no place to go. Normally, the writer will finish writing its data and close the pipe. At this point, the reader sees the EOF (end of file) and executes the request.
Whether or not the “broken pipe” message is issued depends on events at the exact instant the ctrl-C is pressed. If the second cat has just read the x, pressing ctrl-C stops the second cat, pipe1 is closed and the first cat stops quietly, i.e., without a message. On the other hand, if the second cat is waiting for the first to write the x, ctrl-C causes pipe2 to close before the first cat can write to it, and the error message is issued. This sort of random behavior is known as a “race condition”.
Command Substitution
Bash uses named pipes in a really neat way. Recall that when you enclose a command in parenthesis, the command is actually run in a “subshell”; that is, the shell clones itself and the clone interprets the command(s) within the parenthesis. Since the outer shell is running only a single “command”, the output of a complete set of commands can be redirected as a unit. For example, the command:
(ls -l; ls -l) >ls.out
writes two copies of the current directory listing to the file ls.out.
Command substitution occurs when you put a < or > in front of the left parenthesis. For instance, typing the command:
cat <(ls -l) results in the command ls -l executing in a subshell as usual, but redirects the output to a temporary named pipe, which bash creates, names and later deletes. Therefore,cat has a valid file name to read from, and we see the output of ls -l, taking one more step than usual to do so. Similarly, giving >(commands) results in Bash naming a temporary pipe, which the commands inside the parenthesis read for input.
If you want to see whether two directories contain the same file names, run the single command:
cmp <(ls /dir1) <(ls /dir2) The compare program cmp will see the names of two files which it will read and compare. Command substitution also makes the tee command (used to view and save the output of a command) much more useful in that you can cause a single stream of input to be read by multiple readers without resorting to temporary files—bash does all the work for you. The command: ls | tee >(grep foo | wc >foo.count) \
>(grep bar | wc >bar.count) \
| grep baz | wc >baz.count
counts the number of occurrences of foo, bar and baz in the output of ls and writes this information to three separate files. Command substitutions can even be nested:
cat <(cat <(cat <(ls -l)))) works as a very roundabout way to list the current directory. As you can see, while the unnamed pipes allow simple commands to be strung together, named pipes, with a little help from bash, allow whole trees of pipes to be created. The possibilities are limited only by your imagination. http://www.linuxjournal.com/article/2156?page=0,1

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Tips to increase socket (fds) on Linux or Unix systems for network admin

Tips to increase socket (fds) on Linux or Unix systems for network admin
If you ever come across a challenge in your Linux or Unix kernel development, one of the biggest roadblocks is the number of file descriptors (also known as fds or sockets) certain processes have access to. These are obviously set by network administrators with root access in to the box. These same file descriptors can either enhance or degrade the performance of any server using HTTP for the internet. The number you set depends on if the server will be used for either a client or used for something like an email server.
Note that the following instructions can be used by either PAM (pluggable authentication modules) or ssh-ing into a remote server.
You can check the system limits, do:
1. cat /proc/sys/fs/file-max
If you need any changes to surive after rebooting, make changes in /etc/sysctl.conf and edit:
# Maximum number of open files permitted
fs.file-max = 65535

To reload any changes to this file, do: sysctl -p /etc/sysctl.conf.
You can make further changes by following any of these recommendations:
[Find out where __FD_SETSIZE is defined]
% grep “#define __FD_SETSIZE” /usr/include/*.h /usr/include/*/*.h
/usr/include/bits/types.h:#define __FD_SETSIZE 1024
/usr/include/linux/posix_types.h:#define __FD_SETSIZE 1024

Alternately your system may use tabs in which case
something like this should do the trick:
% grep -E “#define\W+__FD_SETSIZE” /usr/include/*.h /usr/include/*/*.h

[Make a local copy of these files]
% cp /usr/include/bits/types.h include/bits/types.h
% cp /usr/include/linux/posix_types.h include/linux/posix_types.h

[Modify them so that they look something like
#define __FD_SETSIZE 65535

[Recompile httperf and/or your server so that it uses a larger file
descriptor set size by using -I include during compliation, this
will allow the compiler/pre-processor to use the new include files
rather than the default versions]

To check and modify limits per shell.
[Using csh: openfiles and descriptors show that the limit here is 1024]
% limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 8146
openfiles 1024

[To increase this to 65535 for all users (as root)]
# vi /etc/security/limits.conf

[Modify or add “nofile” (number of file) entries – note
that a userid can be used in place of *]
* soft nofile 65535
* hard nofile 65535

# vi /etc/pam.d/login
[Add the line]
session required /lib/security/pam_limits.so

[On many systems this will be sufficient – log in as a regular
user and try it before doing the following steps]

[These steps may be required depending on how PAM and ssh are configured
[Note on some systems – Debian?]
# vi /etc/pam.d/ssh
[Note on other systems – RedHat]
# vi /etc/pam.d/sshd
[Add the line]
session required /lib/security/pam_limits.so

# vi /etc/ssh/sshd_config
[May need to modify or add the line]
UsePrivilegeSeparation no

[Restart the ssh daemon]
[Note on some systems – Debian?]
# /etc/init.d/ssh reload
[Note on other systems – RedHat]
# /etc/rc.d/init.d/sshd reload

Also, if you need to do any of these manually. Do in In tcsh
limit descriptors 65535

In bash, do
ulimit -n 65535

Hopefully, these tips will help out any Linux or Unix network administrator. This is also helpful for those wanting to get into either Java or C++ development positions at investment banks.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Linux kernel Unix kernel PID and file descriptor fully explained during network administration

Linux kernel Unix kernel PID and file descriptor fully explained during network administration
One of the more tricky things in Linux within the kernel are the process Ids and file descriptors that go with each binary executable file you run or have access to. All sorts of things can happen within each process a bin executable file, a process could: open the following kind of files:
=> Actual log file
=> /dev files
=> UNIX Sockets
=> Network sockets
=> Library files /lib /lib64
=> Executables and other programs etc
Some handy ways to find out about processes is to use the lsof command. There are number of steps we must accomplish:
1. Get the PID we need by running ‘ps aux|grep prog’ or ‘pidof myprog’. This will then out put the process id. A PID could be 2892
2. We need to list the files open by the PID by using ‘lsof –p 2892’. You could also ‘cd /proc/2892’/fd’. Then ‘ls –l|wc –‘ for the file count used.
Do understand that /proc is file system generated for each process created when the system boots. There is a variety stories from this /proc directory. We get”
? /proc/PID/cmdline : process arguments
? /proc/PID/cwd : process current working directory (symlink)
? /proc/PID/exe : path to actual process executable file (symlink)
? /proc/PID/environ : environment used by process
? /proc/PID/root : the root path as seen by the process. For most processes this will be a link to / unless the process is running in a chroot jail.
? /proc/PID/status : basic information about a process including its run state and memory usage.
? /proc/PID/task : hard links to any tasks that have been started by this (the parent) process.

Hopefully this will give you a better of how processes are managed by the Linux kernel as it executes.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

How do you do debug using primitive Linux or UNIX using GDB? C++ job interview hint

How do you do debug using primitive Linux or UNIX using GDB? C++ job interview hint
GDB is old but I have heard some more advanced interviews do ask these questions so be prepared on how to do the basics within GDB at a shell command line at a Linux terminal session. It is meant for C++ debugging and make sure you set your GCC compile mode to debug to enable this.
As you would, you need to enter on the shell command line:
GDB interaction:
Gdb convdoll
This will load up your biary n execute program. .
break main
This will set breakpoint in your method like main.
run convdoll
Once set, you need execute program the program using ‘run’.
next
This will then move to the next call within your executable file.
print userInput
This will display any variable values GDB may have access to the point at which it is executing.
There are numerous settings in which you would want to run GDB. If you have a process which is being run by your program , it may strangely hang. All you need to do is the get the process id (PID) it is running on. This can be known by executing ‘ps –ef|grep xxx’ where xxx is your program that hung.
gdb convdoll 9198
9198 would obviously be the process id you want to query on.
Sometimes C++ programs that leak memory may terminate abnormally and generate core files. You can always run through the program’s session by debugging it with GDB. Just do on the command line against a dumped core file.
gdb /system/location/convdoll core
Hope this helps you get past those gnarly questions posed at c++ job interview.

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!