The assignment is to create a log collector shell script that reads csv-formatted log events from stdin and publishes them in json format to a remote server via HTTP requests.
The functional requirements are as follows:
-The script is launched by a daemon process which constantly writes event records to stdin of the script.
-The script buffers records until either the buffer size exceeds x records or x seconds has elapsed since first record in the buffer.
-When the buffer is flushed, the records are sent to remote server via HTTP POST request (using curl)
-Input is in csv format (i.e value1,value2,value3 ...). Some fields are transformed using lookup maps and then mapped to a json record (i.e {"field1": "transformedValue1", "field2": "value2", "field3": "transformedValue3"...}). Payload of the POST request is a json array of the records.
-If record mapping fails (i.e field value not found in lookup map when transforming a value), the record is ignored.
-If flushing fails due to network error, the script should keep retrying until it succeeds.
-If flushing fails due to request error (HTTP status other than 200), the script should write error reason to stderr and exit.
-The script should not continue reading from stdin until the buffer is flushed succesfully
-The script accepts remote server address and an authentication token (passed as a header with the POST requests) via command line arguments. Other configurations are defined as constants in the beginning of the script.
Hello,
Not sure if I understood your requirement fully. The base requirement is clear like you need to collect data from a csv and push into a remote http server. That can be easily done - by the way do you have endpoints properly configured to use CURL to send data as HTTP/POST ?
Also, the buffer part is not clear.
By the way, you can refer my profile and if interested please contact me. I am confident in writing shell scripts thats why i bid even though there is some complications in buffer part.
Thank You
Hello!
I can help you to develop a log collector shell script according to your requirements.
As for me I'm a professional system admin with 10+ years of experience. Check my reviews. Recently I've successfully completed several projects where created various bash and shell scripts. See references below:
- script that can automatically install LAMP on Linux servers https://www.freelancer.com/projects/linux/Bash-script-setup-LAMP-stack/
- script that automatically creates proxy servers https://www.freelancer.com/projects/linux/Detailed-Guide-setup-proxies-IPv/ https://www.freelancer.com/projects/php/Squid-Proxy-Script-Multiple-with/ https://www.freelancer.com/projects/linux/Guide-Script-making-rotating-web/ https://www.freelancer.com/projects/linux/Proxy-script-with-luminati/
I'm a real expert on this. I can help you too. Let's discuss the details in private chat. Feel free to contact me.
hi
Can i develope it in php-cgi? It is simple - I can use #!/usr/bin/php or something similar in the start of script and chmod 7 to file - and it will work as shell script - it will be simple for me
about some ideas from Your description - i really didnt understood about websocket of script - how it must work and what to do?
Can we chat about it?
Hello, sir
Your job is exciting me.
Let me do this job for you now just to prove my expertise in this field.
Hope I could awake you interest in me. I am always online, work hard and very sensitive to details.
Thanks for your attention.
Hello
I am working in scripting technologies for 20 years.
I worked in various shells c, korn, bash in various Unix system.
Can you please share the URL and sample log file to parse and output format etc?
Thanks!