Skip to content

Commit

Permalink
Fixed Hadoop groups names
Browse files Browse the repository at this point in the history
  • Loading branch information
slovit committed Oct 24, 2016
1 parent 87701c8 commit 5fac4b0
Show file tree
Hide file tree
Showing 6 changed files with 15 additions and 15 deletions.
6 changes: 3 additions & 3 deletions ansible/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,16 +72,16 @@ During the running of a playbook, our tasks and roles are configured to add new

For the following explanations, we will use the first few lines of the `download.yml` playbook as an example:
```
- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode:download
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode:download
- include: tasks/setup.yml group=hadoop-master:hadoop-worker:download
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker:download
```

* `all_instances` - All hosts from all groups that are either being provisioned and/or used are lumped together into a single host group.
The purpose of this group is so we can write out the hostname and ips of every server to the `/etc/hosts` file of every server. This way,
every server, knows about every other server.

* We do some parsing of host names to generate groups of related servers based on the the way they are named. For example,
in the `hdfs-datanode` group we have the hostnames `dcc-hadoop-worker-[1:2]`. A group called `hadoop_worker` is created containing
in the `hadoop-worker` group we have the hostnames `dcc-hadoop-worker-[1:2]`. A group called `hadoop_worker` is created containing
the hosts `dcc-hadoop-worker-1` and `dcc-hadoop-worker-2`. The utility of this becomes more obvious when you look at groups and playbooks that use
a wider array of servers with different purposes, such as the `portal.yml` playbook with the `portal` host group.

Expand Down
6 changes: 3 additions & 3 deletions ansible/download.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright 2016(c) The Ontario Institute for Cancer Research. All rights reserved.

- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode:download
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode:download
- include: tasks/setup.yml group=hadoop-master:hadoop-worker:download
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker:download

# HDFS - NameNode
- hosts: hadoop_master
Expand Down Expand Up @@ -42,4 +42,4 @@
vars_files:
- "vars/main.yml"
roles:
- download
- download
6 changes: 3 additions & 3 deletions ansible/etl.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright 2016(c) The Ontario Institute for Cancer Research. All rights reserved.

- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode:id-group:etl
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode:id-group:etl
- include: tasks/setup.yml group=hadoop-master:hadoop-worker:id-group:etl
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker:id-group:etl


# HDFS - Namenode
Expand Down Expand Up @@ -85,4 +85,4 @@
- "vars/main.yml"
roles:
- import
- release
- release
4 changes: 2 additions & 2 deletions ansible/hdfs.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright 2016(c) The Ontario Institute for Cancer Research. All rights reserved.

- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode
- include: tasks/setup.yml group=hadoop-master:hadoop-worker
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker

# HDFS - Namenode
- hosts: hadoop-master
Expand Down
4 changes: 2 additions & 2 deletions ansible/spark.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright 2016(c) The Ontario Institute for Cancer Research. All rights reserved.

- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode
- include: tasks/setup.yml group=hadoop-master:hadoop-worker
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker


# HDFS - Namenode
Expand Down
4 changes: 2 additions & 2 deletions ansible/submission.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright 2016(c) The Ontario Institute for Cancer Research. All rights reserved.

#- include: tasks/setup.yml group=hdfs-namenode:hdfs-datanode:submission
- include: tasks/setup-existing.yml group=hdfs-namenode:hdfs-datanode:submission
#- include: tasks/setup.yml group=hadoop-master:hadoop-worker:submission
- include: tasks/setup-existing.yml group=hadoop-master:hadoop-worker:submission

# HDFS - NameNode
- hosts: hadoop_master
Expand Down

0 comments on commit 5fac4b0

Please sign in to comment.