broadcast tensor or index

Question

Hi,

I'm trying to use MPI to parallelize my codes. But I haven't found a proper way to broadcast a tensor or index among workers. Due to the complex structure of tensor in itensor, it's not easy to implement such tensor-broadcast functions efficiently.

Would someone share his/her idea on this issue?

Thanks
Rui-Zhen

Answer 1

Hi Rui-Zhen,
I am actually working on an implementation of real-space parallel DMRG we hope to release publicly later this year. So that will have code inside it that uses MPI to send ITensors between nodes.

To help you out, I just put a file "parallel.h" into ITensor. So if you pull the latest version from GitHub you should see it in the itensor/util/ folder within the ITensor source code. Please treat the codes in there as somewhat experimental, although we have used them a lot in the past so they should be pretty well tested (we used them for the parallel DMRG paper).

The main object you want to use in parallel.h is the Environment object. You should immediately make this object at the top of your "main" function
Environment env(argc,argv);
(passing it argc and argv to create it). Then it will be destructed at the end of your program and will automatically call MPI_Finalize for you. The environment object "env" has a method env.broadcast(T) which takes any object T (such as an ITensor, IQTensor, SiteSet, MPS, MPO and many other types of objects or even plain data like integers) that can be read from or written to a binary stream.

MailBox is another type of object you can make to send to and receive from specific other nodes. You make a MailBox object and give it the id number of the other node (zero-indexed). then you can use the .send and .receive methods to do blocking sends and receives. I don't think it currently supports non-blocking sends and receives.

Finally, to give you more information about your question, every object in the ITensor Library has a .read and .write method that can read/write that object in binary to a stream (either an istream or an ostream). The codes in parallel.h create a stream (in some cases a stringstream, for example) and use the .read or .write methods of an ITensor (or another type of object) to read/write to this stream. Then the stream is sent to another node using a function provided by the MPI library.

So I hope the above paragraph helps you understand that you don't need to know about the "complex structure" of an ITensor or an IQTensor: you just have to use the .read and .write methods to convert an ITensor to or from a binary stream.

Miles

Comments

Hi Miles,

Thanks a lot. I'll try your new codes today.

Rui-Zhen

---

Hi Miles,

"parallel.h" works very well. This is definitely what I want. It's a smart way to broadcast tensor via stringstream. I appreciate a lot.
There is a small mistake in 'parallel.h'. I think we should change obj.read(data) and obj.write(data) to be read(data, obj) and write(data, obj).

And I'd like to share my simple example below. Hope this would help some people.

#include "itensor/all.h"
#include "itensor/util/parallel.h"

using namespace itensor;

int
main(int argc, char *argv[])
    {
    Environment env(argc, argv);

    std::cout << "Worker " << env.rank() << " of " << env.nnodes() << std::endl;


    //
    // Construct and print an ITensor
    //
    auto a = Index("A",5);
    auto b = Index("B",4);

    auto T = ITensor(a,b);
    Print(T);


    env.broadcast(T);


    std::cout << "After broadcast:" << std::endl;
    Print(T);
    

    return 0;
    }

---

Glad it's working well and thanks for posting the helpful example.

About using read(obj,data) etc. you are correct about that. The DoRead and DoWrite helpers in that file are older functions that precede the current read and write functions available in ITensor, so I'll change that.