yadp 0.37

yadp is a distributed application. You can for example, parallelize the processing of large data sets, or, run a task over several hosts. The focus is put both on the parallelization part and on the module which enable the development of the distributed code. The computation task is simplified by parallelizing the program over multiple individual, unreliable, cheap machines. <br /><br />Features:<br /><br /> The source code is distributed, loaded, and executed by workers. Optionally, an input data set can be distributed, but this is not required. Typically, these data are a set of plain text files or database files. From a design point of view, yadp can virtually accept any input file which can be iterated over (nevertheless, nothing is free, for a specific case it must be necessary to implement your own iterator).<br /> yadp is fault tolerant (cf. section below), i.e. one or more agent can fail or stop during its execution without disturbing or breakdown the others agents. Simply, a work part will be re-executed later on an active agent.<br /> An agent must know only one address: the agenda's address. There is not a persistent list of agents hard coded, or something like that, each agent himself declare its activity to the agenda.<br /> When the work is near to be complete, a back-up task mode is started to speed-up the completion: remaining running tasks are duplicated over free workers and executed in parallel. For a same task, the first worker to return the result win, any other result for the same task is now ignored.<br /> All the agents, of all kind can simultaneously be hosted and running on the same machine, so that it can be easy to debug or test.<br /> The agents communicate between them by XML-RPC calls, by using the xmlrpclib module. Each agent implement an XML-RPC server and eventually become client of another server when it needs to request another agent. The resulting benefits are to use a common transport protocol (the http protocol) for exchanging messages, and rely on the XML language for formatting the messages to avoid to write a dedicated parser. Moreover the benefit is huge to use an heavily widespread module instead of writing his own home-made untested (bugged) new (reinvented) module.<br /> Each agent is multi-threaded, one thread per XML-RPC request (the agent use the XML-RPC server for concurrently handling the external requests), and one more thread for processing the internal routines. For example, an applicant agent post a task to a taskscheduler agent by using an XML-RPC call, and the taskscheduler will dispatch this task to a worker agent.<br /> Optionally, and only under Linux (yet), each worker agent can check whether if the system is idle (keyboard, and mouse inactivity) before processing a new task.<br />

Requirements:	No special requirements
Platforms:	*nix, Linux
Keyword:	Application,  Cheap,  Computation,  Development,  Enable,  Individual,  Large,  Machines,  Module,  Multiple,  Parallelization,  Parallelize,  Parallelizing,  Processing,  Program,  Simplified,  Unreliable
Users rating:	0/10