2

I have just installed Google Cloud platform for a free trial. In order to run MapReduce tasks with DataStore, the docs says to run 

./bdutil --upload_files "samples/*" run_command ./test-mr-datastore.sh

But I couldn't get this file on my local and there's a good reason for that, this way to run MapReduce jobs seem to be deprecated see this on github. Is that true, is there an alternative way to create MapReduce tasks from local command lines without requiring BigQuery ?

Qorbani
  • 5,825
  • 2
  • 39
  • 47
epsilones
  • 11,279
  • 21
  • 61
  • 85

2 Answers2

2

The Datastore connector connector is indeed deprecated.

To your question "is there an alternative way to create MapReduce tasks from local command line", one option is to use Google Cloud Dataflow. It's not MapReduce per se, but it's the programming model for parallel data processing which has replaced MapReduce at Google. The Dataflow SDK includes support for Datastore access.

Unlike Hadoop, you don't have to set up a cluster. You just write your code (using the Dataflow SDK) and submit your job from the CLI. The Datastore service will create the needed workers on the fly to process your job and then terminate them.

William Vambenepe
  • 140
  • 4
1

NOTE: Google team removed DataStore connector from bdutil v1.3.0 (2015-05-27) going forward so you might need to use older version or use GCS or BigQuery as proxy to accessing your data in DataStore.

I try to cover as much as I can, but bdutil is require lots more detail which is hard to document it in this answer, but I hope this can give you enough to start:

  • Setup Google Cloud SDK - detail

    # Download SDK
curl https://sdk.cloud.google.com | bash
# Restart your shell
exec -l $SHELL
# Authenticate to GCP
gcloud auth login
# Select Project
gcloud config set project PROJECT_NAME

  • Download and extract bdutil source code that contains DataStore connector. 

    # Download source which contains DataStore connector
wget https://github.com/GoogleCloudPlatform/bdutil/archive/1.2.1.tar.gz -O bdutil.tar.gz
# Extract source
tar -xvzf bdutil.tar.gz
cd bdutil-*/

  • Create bdutil custom environment variable file. Please refer to bdutil configuration documentation about creating correct configuration file, since you need to specify project, number of servers, GCS bucket, machine type, etc...

  • Deploy your Hadoop instances (Full documentation) using datastore_env.sh

    ./bdutil deploy -e YOUR_ENV_FILE.sh,datastore_env.sh

  • Connect to Hadoop Master node

    ./bdutil shell

  • Now in Master node you can run your MapReduce Job which will have access to DataStore as well.

  • Turn down your Hadoop Cluster

     ./bdutil delete
Qorbani
  • 5,825
  • 2
  • 39
  • 47
  • thanks but there is no longer any `datastore_env.sh` file - see when you untar the bdutil file – epsilones Jun 09 '15 at 08:11
  • @Newben, you are absolutely right. They removed it in v1.3.0 (1.3.0 - 2015-05-27 - 12 days ago) - More Detail: https://github.com/GoogleCloudPlatform/bdutil/blob/master/CHANGES.txt – Qorbani Jun 09 '15 at 18:13