Difference between revisions of "Debian 7 Infiniband"

From Supercomputación y Cálculo Científico UIS
Line 1: Line 1:
= Infiniband configuration in Debian 7 =
+
__NOTOC__
  
== Install the required software ==  
+
<div class="thumbnail img-thumbnail">http://wiki.sc3.uis.edu.co/images/a/a8/Logo_sc33.png</div>
 +
<p><div class="btn btn-primary"><i class="fa  fa-long-arrow-left"></i> [[Administración del SC3]]</div></p>
  
{{Command|<nowiki> apt-get -y install infiniband-diags ibutils libmthca ibverbs-utils libibumad libibmad libibverbs1 libmthca1 libmthca1-dbg libmthca-dev libopensm2 opensm </nowiki>}}
+
<div class="column clearfix">
 +
    <div class="col-md-14">
 +
        <div class="well well-midnight">
 +
            <h4></h4>
 +
            <p class="bs"></p>
 +
        </div>
 +
    </div>
 +
</div>
  
 +
<div class="col-md-14">
 +
    <div class="panel panel-darker-white-border">
 +
        <div class="panel-heading">
 +
            <h3 class="panel-title">Infiniband configuration in Debian 7</h3>
 +
        </div>
 +
        <div class="panel-body">
 +
            <p><b>Install the required software</b></p>
 +
            <p>{{Command|<nowiki>apt-get -y install infiniband-diags ibutils libmthca ibverbs-utils libibumad libibmad libibverbs1
 +
libmthca1 libmthca1-dbg libmthca-dev libopensm2 opensm </nowiki>}}
 +
            </p>
 +
            <p><b>Load the kernel Modules</b></p>
 +
            <p>Load the proper linux modules. Edit the file '''/etc/modules''' and add the following lines</p>
 +
            <p>{{File|/etc/modules|<pre><nowiki>
 +
            #tarjeta Mellanox MT26438
 +
            mlx4_ib
  
==Load the kernel Modules ==
+
            #IP over IB
 +
            ib_ipoib
  
Load the proper linux modules. Edit the file '''/etc/modules''' and add the following lines
+
            # Protocol modules
 
+
            # Common modules
{{File|/etc/modules|<pre><nowiki>
+
            rdma_ucm
#tarjeta Mellanox MT26438
+
            ib_umad
mlx4_ib
+
            ib_uverbs
 
+
            </nowiki></pre>}}</p>
#IP over IB
+
            <p>Then, execute the following commands</p>
ib_ipoib
+
            <p>{{Command|<nowiki>modprobe mlx4_ib
 
+
            modprobe ib_ipoib
# Protocol modules
+
            modprobe rdma_ucm
# Common modules
+
            modprobe ib_umad
rdma_ucm
+
            modprobe ib_uverbs </nowiki>}}</p>
ib_umad
+
            <p><b>Subnetting Daemon Configuration</b></p>
ib_uverbs
+
            <p>find the available ports in the card.</p>
</nowiki></pre>}}
+
            <p>{{Command|<nowiki> ibstat -p </nowiki>}}</p>
 
+
            <p>The output are the identifiers of the ports. Example: </p>
Then, execute the following commands
+
            <p>0x78e7d10300244e95</p>
 
+
            <p>Then, edit the file '''/etc/default/opensm'''</p>
{{Command|<nowiki> modprobe mlx4_ib
+
            <p>And change the following line</p>
modprobe ib_ipoib
+
            <p>        PORTS=ALL</p>
modprobe rdma_ucm
+
            <p>Like this</p>
modprobe ib_umad
+
            <p>PORTS="0x78e7d10300244e95"</p>
modprobe ib_uverbs </nowiki>}}
+
            <p><b>Start The Subnetting Service</b></p>
 
+
            <p>Start the opensm service</p>
== Subnetting Daemon Configuration ==
+
            <p>{{Command|<nowiki> /etc/init.d/opensm start </nowiki>}}</p>
 
+
            <p><b>Configure the Infiniband Interface</b></p>
find the available ports in the card.
+
            <p>Edit the file '''/etc/network/interfaces''' and add the following lines</p>
 
+
            <p>{{File|/etc/network/interfaces|<pre><nowiki>
{{Command|<nowiki> ibstat -p </nowiki>}}
+
            auto ib0
 
+
            iface ib0 inet static
The output are the identifiers of the ports. Example:  
+
            address 192.168.38.71
 
+
            netmask 255.255.255.0
        0x78e7d10300244e95
+
            broadcast 192.168.38.255
 
+
            </nowiki></pre>}}</p>
Then, edit the file '''/etc/default/opensm'''
+
            <p>Then, start the interface</p>
 
+
            <p>{{Command|<nowiki> ifup ib0 </nowiki>}}</p>
And change the following line
+
            <p>Test the interface using the following commands</p>
 
+
            <p>{{Command|<nowiki> ibstat
        PORTS=ALL
+
            ibhosts </nowiki>}}</p>
 
+
        </div>
Like this
+
    </div>
 
+
</div>
        PORTS="0x78e7d10300244e95"
 
 
 
== Start The Subnetting Service ==
 
 
 
Start the opensm service
 
 
 
{{Command|<nowiki> /etc/init.d/opensm start </nowiki>}}
 
 
 
== Configure the Infiniband Interface ==
 
 
 
Edit the file '''/etc/network/interfaces''' and add the following lines
 
 
 
 
 
{{File|/etc/network/interfaces|<pre><nowiki>
 
auto ib0
 
iface ib0 inet static
 
address 192.168.38.71
 
netmask 255.255.255.0
 
broadcast 192.168.38.255
 
</nowiki></pre>}}
 
 
 
Then, start the interface
 
 
 
{{Command|<nowiki> ifup ib0 </nowiki>}}
 
 
 
Test the interface using the following commands
 
 
 
{{Command|<nowiki> ibstat
 
ibhosts </nowiki>}}
 

Revision as of 11:00, 9 April 2015


Logo_sc33.png

Infiniband configuration in Debian 7

Install the required software

apt-get -y install infiniband-diags ibutils libmthca ibverbs-utils libibumad libibmad libibverbs1 libmthca1 libmthca1-dbg libmthca-dev libopensm2 opensm

Load the kernel Modules

Load the proper linux modules. Edit the file /etc/modules and add the following lines

File: /etc/modules
            #tarjeta Mellanox MT26438
            mlx4_ib

            #IP over IB
            ib_ipoib

            # Protocol modules
            # Common modules
            rdma_ucm
            ib_umad
            ib_uverbs
            

Then, execute the following commands

modprobe mlx4_ib modprobe ib_ipoib modprobe rdma_ucm modprobe ib_umad modprobe ib_uverbs

Subnetting Daemon Configuration

find the available ports in the card.

ibstat -p

The output are the identifiers of the ports. Example:

0x78e7d10300244e95

Then, edit the file /etc/default/opensm

And change the following line

PORTS=ALL

Like this

PORTS="0x78e7d10300244e95"

Start The Subnetting Service

Start the opensm service

/etc/init.d/opensm start

Configure the Infiniband Interface

Edit the file /etc/network/interfaces and add the following lines

File: /etc/network/interfaces
            auto ib0
            iface ib0 inet static
            address 192.168.38.71
            netmask 255.255.255.0
            broadcast 192.168.38.255
            

Then, start the interface

ifup ib0

Test the interface using the following commands

ibstat ibhosts