Script to analyze huge set of data

Hi, I have seen your project details. I will be very glad to do this job. I guarantee deadline commitment. I hope to discuss more details with you soon...

HI , i have done similar work of combing various CSV and Excel Files and make it into 1 file. Basically we select / break the file into picese taking 100 k every time and transfer etc. Thank you

Hi, You are saying that you get an "out of memory" exception. Could you tell me how large at the individual Excel files? What script and/or programming language are you using at the moment? If the individual Excel files are very large then out of memory exception can be avoided by not loading the entire file in memory but processing it gradually. Could you tell me if the Excel files are .xls (Excel 2003 and earlier) or .xlsx (Excel 2007 and later)? Low speed is usually caused by low memory available (thus it is likely a direct consequence of the way files are processed) as Windows is forced to move memory from various processes to the page file which significantly slows down the OS. Thus avoiding high memory usage should solve this as well. Which version and edition of SQL Server are you using? What query do you use to import data? I need to understand your process in order to be able to tell how to address the performance issues. Regards, Jecho

Hi , Kindly refer below details . Sr. Software Professional having 7.5+ years experience. ****************************************************** ****************************************************** Technology : Windows Application Web Application Mobile Application ****************************************************** Skills : C# .Net, MVC , PHP, VB .Net, HTML, CSS, Javascript, Jquery, Anguler Js, Mobile Jquery,WPF, WCF,WEBAPI, PHP, Apache , Android, IOS, nopCommerce, Magento, suiteCRM ****************************************************** Database Language : MSSQL, Oracle, MySQL, Microsoft Access ****************************************************** Reporting Langauge : Crystal Reports, RDLC, Syncfusion View, Pivot, Chart Reports , DevExpress, SSRS Reporting Tool : MIS Reports, Chart Reports, Stock Reports, Inventory Reports. ****************************************************** Thanks & Regards, Ketan H Patel Sr. Software Professional

Hello Sir/Madam, I am having around 13 years of experience in mobile & .NET technologies(C#/VB). I am expert in asp.net MVC, Cordova, Xamarin, Android, IPhone, Web API, asp.net, WPF, AngularJs, TypeScript, ExtJS, YUI, JQuery, Bootstrapper frameworks. I have gone through your requirement & it exactly match with my skills. Looking forward a positive response from your end

hello. 15 GB is size of 1 file ? or all files combined ? please reply. I can write a c# app to handle them. thanks

Hi I have been in this industry for 5 years and such jobs are my daily practice.I can assure you that if you work with me once, you will always work with me for these kind of projects. I will do it with in your budget $500.

I will write a program which will take all files one by one and insert into database. I am having more than 8 years of experience in IT industry.I have worked on .net(c#.net,vb.net,asp,asp.net,mvc) with sql server 2005/2008/2012/2014/2016 and oracle(pl/sql). I can provide you better solution by utilizing my knowledge and skills.

Dear Employer, I do a lot of projects using C#. All the projects that I have done are using C# only. I will give you a non stop around the clock service. Please try me once. Thanks and Regards, Ashish Singhal

Greetings sir, i am an expert freelancer for this job and your 100% satisfaction is assured if you allow me to serve. Here is the reason. Why you should pick me? a) I am a very expert and have the same kind of experience of 5 years. b) I work very hard (16+ hours a day and 7 days a week) and also very fast so... it will be done very soon than most of the other providers c) And most important part is my policy: "I will give you (to my client) life time support (as long as you keep relation with me). And fix any bugs/problem without any cost. So, don't ever worry about me” Please sir, leave a reply ASAP, as I am waiting for your kind reply

I want to discuss this project with you further, let me know the best suitable time for you to schedule the meeting, Feel free to message me at any time, i used to be online 14 hrs in a day on this website so probably you will get a quick response from my end.

To avoid out of memory - you need to not store data in data structures but process them as a stream row by row. If you need to remove duplicates, hash function on the row would do its job (just need to store hashes in memory to find duplicates, not all the data). Very slow speed - it depends on machine, libraries and algorithms you are using.

Hi, I've read the description. To avoid the described bottlenecks I would do the following: - Since I expect it to be inserted/update operations, I would process the whole bulk of files by multiple threads, where each thread would handle one file at a time. I would optimize the number of threads depending on the architecture of the machine. This should improve the speed the files are being processed. - As for the memory limit, each Thread will only keep a certain amount (such as 100 file rows) in its memory. After this memory (or buffer) is filled up, it executes query against the database and then loads up its memory again. This way you can control, how much memory your program uses and not end up with an OutOfMemoryException being thrown. I'll write it for 500$ in C#, MS SQL. Let me know if you have any further questions, Thank you, Pavol

Hello, I can write script in C# and assure you that you will get the best quality work from my side. You can get hold of me instantly to discuss this position further. I am available via. Skype, E-mail, Whatsapp. Looking ahead to working with you. Regards, Bhanu

Hi sir...!! I am professional java developer and will do that task easily and have done similar tasks before so contact me if you are looking quality work then do contact me Thank you. Will do it in a day. with a negotiatable budget.

Dear Sir, I have more than 11 years experience of working in Microsoft Technologies; especially ASP.NET (C#, VB.NET) and SQL Server). I have experience in a variety of applications, these applications involved startups as well as maintenance of existing applications. I have recently worked on a similar kind of module, where we have a number of monthly excel sheets which are required to be imported into the DB. I would like to know, if you want to build your system as a web application or a desktop application, and whether will there be any primary/unique key in the data with the help of which we can identify duplicates. Please respond to my bid so that I can send you my resume via freelancer.com chat / message. Looking forward to hear a positive response from you soon. Best Regards, Zubair Ahmed Khan

I can do it by C#, SQL server 2014 and Crystal Report. Please send me link and other requirements for clear concept.

Please share the sample file now and I will show you the demo tomorrow. This will be an desktop based exe file that you will run. Codes will be yours so you will understand the theory.

Hi Pedro here, I am really interested to work on this project, I can start the work now , and can provide the best services from my end. Please come on chat to discuss more about the project. Thanks!

Hi, I (Myself a senior PHP Developer) have a team having experience of more than 10 years in developing web and mobile applications. I have a team of expert programmers who write robust and bug-free codes at the very first instance. After seeing our quality of work and the cooperation offered to you even after the work has been delivered to you, you’ll come to us again and again to avail our services. I read through the job details extremely carefully and I am absolutely sure that I can do the project very well. I can achieve the results that you are asking for within your timeframe and budget. I am confident I can exceed your expectations and I would love to work on your project. Warm Regards ~Team Shopify

I can do this for you very quickly. In fact i'm doing similar work right now and can show if you if you like. Ping me over chat to discuss further.