Limits on the cuSparse tridiagonal solver?

Question

I have decided to rewrite one of my serial codes to CUDA. A large section of the code is to invert a large tridiagonal matrix for differing right hand sides. I then came across cusparseSgtsv from the cuSparse library. I got a sample code to work for small matrices, but when the matrix size got above 1024, nothing but nan. Did I miss something in the documentation?

Here is the sample code. For N=1024, the code works fine. For N=1025, it is nan all the way down.

#include<iostream>                                                                                                                              
#include<cuda_runtime.h>                                                                                                                        
#include<cusparse_v2.h>                                                                                                                         

using namespace std;

__global__ void assignMat(float *a,float *b,float *c,float *r)
{                                                             
    int tid=threadIdx.x+blockDim.x*blockIdx.x;            

    a[tid]=0;
    b[tid]=1;
    c[tid]=0;
    r[tid]=tid;
}

int main()
{
    float *d_a,*d_b,*d_c,*d_r;
    float *h_r;
    int N=1025;
    cusparseStatus_t status;
    cusparseHandle_t handle=0;

    status=cusparseCreate(&handle);

    h_r=(float *)malloc(N*sizeof(float));
    cudaMalloc((void **)&d_a,N*sizeof(float));
    cudaMalloc((void **)&d_b,N*sizeof(float));
    cudaMalloc((void **)&d_c,N*sizeof(float));
    cudaMalloc((void **)&d_r,N*sizeof(float));

    assignMat<<<1,N>>>(d_a,d_b,d_c,d_r);
    status=cusparseSgtsv(handle,N,1,d_a,d_b,d_c,d_r,N);
    if (status != CUSPARSE_STATUS_SUCCESS)
    {
            cout << status << endl;
    }
    else
    {
            cudaMemcpy(h_r,d_r,N*sizeof(float),cudaMemcpyDeviceToHost);
            for (int i=0;i<N;i++)
                    cout << i << " " << h_r[i] << endl;
    }

    free(h_r);
    cudaFree(d_a);cudaFree(d_b);cudaFree(d_c);cudaFree(d_r);
}

Answer 1

Did I miss something in the documentation?

Not in the cuSparse documentation, no.

However, there is a hard limit on the number of threads per block, so your assignMat kernel stops working once N > 1024. You can read about how to select legal kernel launch parameters here. If your code contained error checking or you ran the program with cuda-memcheck, you probably would have been able to detect the problem yourself at runtime.