How about setting the task priority of the driver to a value larger than 50?
e.g.
struct sched_param PARAM = {.sched_priority = MAX_RT_PRIO };
PARAM.sched_priority = 51;
sched_setscheduler(task, SCHED_FIFO, &PARAM);
Another way is to use local_irq_disable() and local_irq_enable() to stop the interrupt from other priority when driver trying to read the data.