Copy Data Files from AWS EC2 to S3

Goal: To copy files from a EC2 instance running Windows to S3.

For example, you may want to copy database backup files from a Windows Server with SQL Server running in a AWS server instance.

Approaches:

1. Write your own method using AWS SDK for .NET

http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpNET.html

2.  Use an open source Windows command utility

http://s3.codeplex.com/wikipage?title=put%20command&referringTitle=Documentation

3. Use powershell to call upon a free 3rd party snap-in

http://friism.com/ec2-sql-server-backup-strategies-and-tactics

For example, cloudberry has this free snap-in:

http://www.cloudberrylab.com/default.aspx?page=amazon-s3-powershell

Note that CloudBerry Explorer for Amazon S3 is Freeware.

My Approach:

On Amazon side:

  1. Using Amazon Management Console, I go to IAM Management
  2. Create a group with S3 full access permission. 
  3. Create a user and add to that group.
  4. Go to S3.  Create a new bucket (e.g. mysqlbackup)
  5. Note that there is no need to add permission to the bucket.  You cannot add a IAM account as a grantee.

On the SQL Server:

  1. Download s3.exe from here and copy it to a folder (e.g. C:\Program Files\WinWin\)
  2. Create a new job “CopyDatabaseBackupToS3”
  3. Create a new step.
  4. For Type, choose “Operating system (CmdExec)”
  5. Under the command textbox, enter

    “C:\Program Files\WinWin\s3.exe” put mysqlbackup C:\Backup\ /sub:withdelete /yes /sync /nogui”

  6. Create a daily maintenance plan with the following steps 
    1. Backup Database Task. 
    2. Maintenance Cleanup Task (optional: to delete old database files)
    3. Step 3: Execute SQL Server Agent Task (to execute  CopyDatabaseBackupToS3)

Explanation on the s3.exe command syntax:

In the above example, s3.exe is located at “C:\Program Files\WinWin\”

put – put (e.g. store files) to S3

mysqlbackup – the name of the S3 bucket to put files into

C:\Backup\ – this is the source directory where all my sql backup files are stored.

/sub:withdelete – this copy the entire directory tree and also delete keys on S3 that correspond to a local file

/yes – when used with /sub:withdelete, it suppresses prompting on each delete.

/sync – only uploads new or modified files since last upload.

/nogui – suppress windows popup.

But what about authentication?

You need to either use s3.exe auth or you can save it to Windows user profile.  If you run the s3.exe in a command line under the service account, you will be prompted for the access key Id and Secret Access key.

image

It is recommended that you encrypt your Secret Access Key with a password.

Advertisements

Memory Repair

http://www.technologyreview.com/featuredstory/515981/repairing-bad-memories/

Excerpts:

“… The results of human experiments indicating that memories are reshaped and rewritten every time we recall an event. And, the research suggested, if mitigating information about a traumatic or unhappy event is introduced within a narrow window of opportunity after its recall—during the few hours it takes for the brain to rebuild the memory in the biological brick and mortar of molecules—the emotional experience of the memory can essentially be rewritten.

“When you affect emotional memory, you don’t affect the content,” Schiller explains. “You still remember perfectly. You just don’t have the emotional memory.”

“In the human experiments, volunteers were shown a blue square on a computer screen and then given a shock. Once the blue square was associated with an impending shock, the fear memory was in place. Schiller went on to show that if she repeated the sequence that produced the fear memory the following day but broke the association within a narrow window of time—that is, showed the blue square without delivering the shock—this new information was incorporated into the memory.

Here, too, the timing was crucial. If the blue square that wasn’t followed by a shock was shown within 10 minutes of the initial memory recall, the human subjects reconsolidated the memory without fear. If it happened six hours later, the initial fear memory persisted. Put another way, intervening during the brief window when the brain was rewriting its memory offered a chance to revise the initial memory itself while diminishing the emotion (fear) that came with it. By mastering the timing, the NYU group had essentially created a scenario in which humans could rewrite a fearsome memory and give it an unfrightening ending. And this new ending was robust: when Schiller and her colleagues called their subjects back into the lab a year later, they were able to show that the fear associated with the memory was still blocked.”

“… memory is best preserved in the form of a story that collects, distills, and fixes both the physical and the emotional details of an event. “The only way to freeze a memory,” she says, “is to put it in a story.””