Friday, 9 October 2015

How to create a large file using shell script, To test with Hadoop Map Reduce processing

1. Open new Terminal.

2. Create ''  file using below command:


3. Create 'smallfile.txt' file using below command:

touch smallfile.txt

4. Open the file using below command:

gedit smallfile.txt

5. Enter some sample data into the file.

I am going to hyderabad
I am learning hadoop course

6. Save the file using command: Ctrl + S

7. Now prepare a large file using below steps.

8. Open  the file using below command:


9.  Add the below script into the file.

for i in {1..1000}
   cat smallfile.txt >> largefile.txt

10. Save the file using command: Ctrl + S

11. Provide execute permission to script file using below command:

chmod +x

12. Run the script file to generate a large file using below command:


13. Verify the 'largefile.txt' in current folder.

14. Run the above command multiple times to get more large file.

15. Try "hadoop fs -put " commad to place into HDFS.

16. Run Mapreduce programs using above file in step15.

Related Posts Plugin for WordPress, Blogger...