DATA TRANSFER FROM MYSQL DATABASE TO MS SQL DATABASE

Contents:

1. 2. 3.

4.

Prologue Terminology used Setting Up Machine 1 a. Setting up MySQL Server for exporting data b. Installing WinSCP on machine 1 c. Creating a scheduled task on machine 1 Setting up Machine 2 a. Installing and configuring freeSSHd on machine 2 b. Creating database and tables in MS SQL server on machine 2 c. Setting up MS SQL Server for importing data

PROLOGUE
We are gathering huge amount of data from Anasage website visitors, like which page are they viewing from our website, how much they are spending, which visitors is most active etc. But cannot process this data unless we transfer it to much more accessible database where we can perform data mining other required operations on it. So in this document I am going to set-up a system using which we will transfer this data from MySQL database of Piwik Web Analytics tool to MS SQL Server running on a remote machine. Before continuing please make sure that you have access to all the required files and soft-wares. SQL files are stored in TFS server under Document Analytics folder.

Terminology Used In This Manual

Terms used in this manual are explained below: MACHINE 1: This is the machine which is running MySQL Server and it holds data tracked by Piwik Web Analytics Tool from Anasage website visitors. This machine will act as SSH client during transfer of CSV files to machine 2 (explained below). MACHINE 2: This is the machine which is running MS SQL Server and our tracked data from MySQL database will be transferred to MS SQL Server here. In transferring CSV files from machine 1 to machine 2 this machine will act as SSH Server.

SETTING UP MACHINE 2
I am assuming that MS SQL Server 2008 R2 is already installed on machine 2. If it is not then you can consult its installation manual prepared by Waseem. INSTALLING AND CONFIGURING FREESSHD ON MACHINE 2 We will first install freeSSHd software on this machine. This software will act as SFTP/SSH server so that client running on machine 1 can connect to it. 1. From Extra software folder run freeSSHd.exe.

2.

Click Yes. And then click Next on next window.

3.

Keep clicking next until you reach install screen. There just click install and you are almost done. Click close in next screen:

4.

Next you will be asked to create private key. Select Yes:

5.

In next screen click Yes. This will make freeSSHd run in background as a service, so that even if its Interface is not running we can still connect to it. NOTE: if you installed freeSSHd already and it did not work please select NO here. It will solve the problem but you will have to start this application everytime

server restarts.

6.

A command prompt screen will appear for a second and then following window will appear: select Finish.

7.

This software is installed. Now we need to make some configurations in it. Right Click FreeSSHd on desktop to run this program as an administrator. Click OK. A new icon will appear in tray area of task bar. Click on its icon.

8.

A new window will appear on desktop.

9.

Create a new folderin C:\ drive named remote_ftp_folder. Then go to SFTP tab on freeSSHd Setting window select this folder as home folder for SFTP. Our CSV files transferred by client machine 1 will be

stored here.

10. Next go to SSH tab and make sure Listen Address matches IP address of machine 2.

11. Create a new folder Public Keys inside remote_ftp_folder installation directory, Copy file user_key to this directory.

12. Go to Authentication Tab and updates settings as follow:

13. Next click Users tab and select Add.

14. Enter user name as shown in above picture. Make sure you entered these values as it is and case of every letter is same. Also Authorization should be selected as Public Key. Click OK.

15. Next go to logging tab and entered following

16. Next go to Automatic Update and turn off check box for Check for newer version on startup. Next click apply from below. You will be presented with a message box.

17. Click Yes and then click OK from bottom navigation bar. CREATING DATABASE AN D TABLES IN MS SQL SERVER ON MACHINE 2 Now we will create a database in MS SQL Server and create tables in it.

1.

First run Microsoft SQL Server Management Studio.

2.

Enter your credentials and Select Connect.

3.

Press Ctrl+O to open file and select database.sql from folder Machine 2-> SQL Server Tables

4.

Then press F5 to execute query. Repeat this step again with files actions_table.sql, site_table.sql, visit_actions_table.sql and visit_table.sql. And execute all of them. This will create four tables inside this database.

SETTING UP MS SQL SERVER FOR IMPORTING DATA 1. Now we need to create stored procedure in this database to import data from CSV files and insert it into tables. Press Ctrl+O to open stored procedure file this file is located under folder Machine 2-> Stored Procedure->ImportTrackingData.sql. After opening it just execute it. Now one last step is remaining: Creating SQL Agent Job to execute this stored procedure at regular intervals to import data from CSV files. Right Click on SQL Server Agent-> Jobs and select create new job.

2. 3.

4.

Enter following information in window which will appear:

5.

Go to Steps from left side-bar and then click new. Enter information as follows: (Text inside Command section is [tracking_data_db].[dbo].[import_tracking_data])

6.

Next go to Advanced from left side bar and make sure all values are as follows: Then press OK.

7.

Next Click Schedules and then click new. Input information as follows. And Press ok.

8.

Press ok to create this job.

9.

Now one last step is remaining. Create a folder named formats inside C:\remote_ftp_folder we previously create and move these four file to it from Machine 2->MS SQL Format Files folder.

SETTING UP MACHINE 1
SETTING UP MYSQL SER VER FOR EXPORTING DA TA Setting up MySQL Server for exporting data involves following steps: 1. 2. Creating a stored procedure to export data Creating an event to execute stored procedure regularly at specific intervals

SQL File for creating a stored procedure and event can be found under directory Machine 1 ->MySQL Files. 1. First make sure that wamp server is running and it is online. Steps to start wamp server are:

a.

Click Start Menu and type wamp. Select start WampServer.

2.

b. Then in tray click on and click Restart all Services. Now start your browser and go to this address: http://localhost/phpmyadmin/. You will see following page:

3.

Enter credentials and click Go.

4.

Now click on piwik_db from left side-bar. You should see following page:

5.

Click Import from top menu-bar.

6. 7. 8.

Click Browse and select file Export Tracked Data.sql and click G0. You will see following messeage :

Import has been successfully finished, 2 queries executed. (Export Tracked Data.sql)
Now click Import and select Start Procedure Event.sql and click Go. You will see following message:

Import has been successfully finished, 2 queries executed. (Export Tracked Data.sql)
Now stored procedure and event both are created in MySQL database.

SETTINGUP SSHCLIENT ON MACHINE 1 Next we will install a SSH client on this machine to transfer CSV files to machine 2. 1. Create a folder piwik_tracking_data inside drive where wamp server is installed like C:\ drive.

2.

Copy scripts folder from Machine 1->SSH Client to this folder.

3.

This script folder contains a custom built SSH client which uses SharpSSH at its backend. Now go inside scripts folder and open SSHClient.exe.xml in your favorite text editor.

4.

5.

This file contains settings for our SSH server and files to be transferred. Enter correct values Also if you need to add more files make sure to append key name with correct number like file1, file2 etc. Also do not remove any key from this configuration file otherwise this client may not work properly. Save the file and close text editor.

CREATING A SCHEDULED TASK ON MACHINE 1 1. Now we will create a scheduled task in maschie 1 to run SSHClient.exe file at regular intervals.

2.

Go to Control Panel and click on Administrative Tools.

3.

Double click on Task Scheduler.

4.

You should see a window like this:

5.

Click on Create Task from right side-bar. And fill up the form which will appear as follows:

6.

Now go to Triggers tab.Click New enter following data. Then click Ok once.

7.

Next go to Actions tab. Click New and enter following data. Note that text entered in Start In field is path to SSHClient.exe file e.g. in my case it is C:\piwik_tracking_data\scripts. Click Ok.

8.

Now go to Conditions tab and make sure your selections match this image:

9.

Next go to Settings tab and repeat previous step.

10. Now click OK. Enter you credentials if it asks:

11. Close Task Sheduler. Now MACHINE 1 is completely setup. Now this is it. Whole setup should be working fine now.