	*************************************************
	*                                               *
	*          ONE-View report generation           *
	*                                               *
	*************************************************

[MAQAO] Info: Experiment configuration summary is available adding -dbg=1 in command line

* [MAQAO] Warning: Experiment directory /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/run/oneview_runs/defaults/icx/oneview_results_1786614283 already exists and is reused.
           It can be replaced using --replace in the command line.
[MAQAO] Info: 
[MAQAO] Info: START THE APPLICATION PROFILING
[MAQAO] Info: -> RUNNING THE PROFILER...
[MAQAO] Info:   LPROF has already been run
[MAQAO] Info: STOP THE APPLICATION PROFILING
[MAQAO] Info: 
[MAQAO] Info: START FUNCTIONS AND LOOPS ANALYSIS ...
[MAQAO] Info: STOP FUNCTIONS AND LOOPS ANALYSIS ...
[MAQAO] Info: 
[MAQAO] Info: START THE REPORT GENERATION
[MAQAO] Info: -> ONE-VIEW EXPERIMENT DIRECTORY: /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/run/oneview_runs/defaults/icx/oneview_results_1786614283


+====================================================================================================================+
+                                                    1  -  GLOBAL                                                    +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                             1.1  -  Experiment Summary                                             +
+--------------------------------------------------------------------------------------------------------------------+

  Application:			/beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/run/oneview_runs/defaults/orig/exec
  Timestamp:			2026-08-13 11:44:42
  Universal Timestamp:		1786614282
  Experiment Type:		MPI; OpenMP; Throughput; 
  Machine:			isix06.benchmarkcenter.megware.com
  Architecture:			x86_64
  Micro Architecture:		GRANITE_RAPIDS
  Model Name:			Intel(R) Xeon(R) 6972P
  Cache Size:			491520 KB
  Number of Cores:		96
  OS Version:			Linux 5.14.0-687.31.1.el9_8.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Aug 1 05:38:01 EDT 2026
  Compilation Options:		
		exec: Intel(R) Fortran 26.0-1156 
  Number of processes observed:	6
  Number of threads observed:	192
  MAQAO version:		2026.1.0
  MAQAO build:			6d1be1d51c1e63266254997eb301734a7264775d::20260810-150026




+--------------------------------------------------------------------------------------------------------------------+
+                                               1.2  -  Global Metrics                                               +
+--------------------------------------------------------------------------------------------------------------------+

  Total Time:				43.91 s
  Max (Thread Active Time):		43.28 s
  Average Active Time:			38.06 s
  Activity Ratio:			99.9 %
  Average number of active threads:	166.435
  Affinity Stability:			99.5 %
  Time spent in analyzed loops:		94.6 %
  Time spent in analyzed innermost loops: 94.6 %
  Time spent in user code:		94.6 %
  Compilation Options Score:		16.67
  Array Access Efficiency:		91.1 %

   Potential Speedups
  ----------------------------------------------------
  Perfect Flow Complexity:		1.00
  Perfect OpenMP/MPI/Pthread/TBB:	1.11
  Perfect OpenMP/MPI/Pthread/TBB + Load Distribution:	1.19
  If No Scalar Integer:
      Potential Speedup:		1.00
      Nb Loops to get 80%:		1
  If FP Vectorized:
      Potential Speedup:		1.12
      Nb Loops to get 80%:		11
  If Fully Vectorized:
      Potential Speedup:		1.30
      Nb Loops to get 80%:		14
  If Only FP Arithmetic:
      Potential Speedup:		1.06
      Nb Loops to get 80%:		6




+--------------------------------------------------------------------------------------------------------------------+
+                                             1.3  -  Potential Speedups                                             +
+--------------------------------------------------------------------------------------------------------------------+

  If No Scalar Integer:
      Number of loops   | 1      | 9      | 18     | 26     | 35     | 
      Cumulated Speedup | 1.0010 | 1.0012 | 1.0012 | 1.0012 | 1.0012 | 
  Top 5 loops:
    exec - 131:	1.001
    exec - 224:	1.0011
    exec - 333:	1.0011
    exec - 339:	1.0012
    exec - 142:	1.0012

  If FP Vectorized:
      Number of loops   | 1      | 9      | 18     | 26     | 35     | 
      Cumulated Speedup | 1.0165 | 1.0855 | 1.1185 | 1.1185 | 1.1185 | 
  Top 5 loops:
    exec - 67:	1.0165
    exec - 213:	1.0309
    exec - 131:	1.0414
    exec - 144:	1.0491
    exec - 137:	1.0567

  If Fully Vectorized:
      Number of loops   | 1      | 9      | 18     | 26     | 35     | 
      Cumulated Speedup | 1.0212 | 1.1571 | 1.2805 | 1.2985 | 1.2985 | 
  Top 5 loops:
    exec - 213:	1.0212
    exec - 67:	1.0383
    exec - 144:	1.055
    exec - 137:	1.0717
    exec - 133:	1.0888

  If Only FP Arithmetic:
      Number of loops   | 1      | 9      | 18     | 26     | 35     | 
      Cumulated Speedup | 1.0138 | 1.0618 | 1.0621 | 1.0621 | 1.0621 | 
  Top 5 loops:
    exec - 83:	1.0138
    exec - 131:	1.0241
    exec - 139:	1.0332
    exec - 142:	1.0422
    exec - 74:	1.0475



+====================================================================================================================+
+                                                   2  -  SUMMARY                                                    +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                             2.1  -  EXPERIMENT QUALITY                                             +
+--------------------------------------------------------------------------------------------------------------------+

  [4 / 4] Application profile is long enough (43.28 s)
To have good quality measurements, it is advised that the application profiling time is greater than 10 seconds.

  [3 / 3] Most of time spent in analyzed modules comes from functions with source/debug info
-g option gives access to debugging informations, such are source locations.

  [0 / 3] Architecture specific options are unknown for some functions (compilation options info are not available)
Architecture specific options are needed to produce efficient code for a specific processor ( -march=(target) ).
Architecture specific options are needed to produce efficient code for a specific processor ( -x(target), -ax(target)
or -march=(target)).

  [0 / 3] Most of time spent in analyzed modules comes from functions without compilation options informations
Functions without compilation options information cumulate 94.76% of the time spent in analyzed modules. For exec,
check that both -g and -grecord-gcc-switches are present. Remark: if such options are indeed used, this can also be
due to some compiler built-in functions (typically math) or statically linked libraries. This warning can be ignored
in that case.

  [0 / 3] Optimization level is unknown for some functions (compilation options info are not available)
To have better performances, it is advised to help the compiler by using a proper optimization level (-O2 of higher).
Warning, depending on compilers, faster optimization levels can decrease numeric accuracy.

  [3 / 3] Host configuration allows retrieval of all necessary metrics.


  [2 / 2] Application is correctly profiled ("Others" category represents 0.13 % of the execution time)
To have a representative profiling, it is advised that the category "Others" represents less than 20% of the execution
time in order to analyze as much as possible of the user code

  [1 / 1] Lstopo present. The Topology lstopo report will be generated.



+--------------------------------------------------------------------------------------------------------------------+
+                                                2.2  -  CODE QUALITY                                                +
+--------------------------------------------------------------------------------------------------------------------+

  [4 / 4] Enough time of the experiment time spent in analyzed loops (94.64%)
If the time spent in analyzed loops is less than 30%, standard loop optimizations will have a limited impact on
application performances.

  [3 / 4] A significant amount of threads are idle (13.32%)
On average, more than 10% of observed threads are idle. Such threads are probably IO/sync waiting. Some hints: use
faster filesystems to read/write data, improve parallel load balancing and/or scheduling.

  [4 / 4] CPU activity is good
CPU cores are active 99.86% of time

  [4 / 4] Loop profile is not flat
At least one loop coverage is greater than 4% (6.24%), representing an hotspot for the application

  [4 / 4] Enough time of the experiment time spent in analyzed innermost loops (94.59%)
If the time spent in analyzed innermost loops is less than 15%, standard innermost loop optimizations such as
vectorisation will have a limited impact on application performances.

  [4 / 4] Affinity is good (99.47%)
Threads are not migrating to CPU cores: probably successfully pinned

  [3 / 3] Less than 10% (0.00%) is spend in BLAS1 operations
It could be more efficient to inline by hand BLAS1 operations

  [0 / 3] Too many functions do not use all threads
Functions running on a reduced number of threads (typically sequential code) cover at least 10% of application
walltime (15.58%). Check both "Max Inclusive Time Over Threads" and "Nb Threads" in Functions or Loops tabs and
consider parallelizing sequential regions or improving parallelization of regions running on a reduced number of
threads

  [3 / 3] Cumulative Outermost/In between loops coverage (0.05%) lower than cumulative innermost loop coverage (94.59%)
Having cumulative Outermost/In between loops coverage greater than cumulative innermost loop coverage will make loop
optimization more complex

  [2 / 2] Less than 10% (0.00%) is spend in BLAS2 operations
BLAS2 calls usually could make a poor cache usage and could benefit from inlining.

  [2 / 2] Less than 10% (0.00%) is spend in Libm/SVML (special functions)



+--------------------------------------------------------------------------------------------------------------------+
+                                               2.3  -  LOOPS OVERVIEW                                               +
+--------------------------------------------------------------------------------------------------------------------+

  Top 5 loops:
   + exec - 61  :
     analysis: Execution Time: 6 % - Vectorization Ratio: 100.00 % - Vector Length Use: 50.00 %
     Loop Computation Issues: 8
        [8] [SA] Presence of expensive FP instructions - Perform hoisting, change algorithm, use SVML or proper
            numerical library or perform value profiling (count the number of distinct input values). There are 2
            issues (= instructions) costing 4 points each.
     Data Access Issues: 2
        [0] [SA] Inefficient vectorization: more than 10% of the vector loads instructions are unaligned - When
            allocating arrays, don’t forget to align them. There are 0 issues ( = arrays) costing 2 points each
        [2] [SA] More than 20% of the loads are accessing the stack - Perform loop splitting to decrease pressure on
            registers. This issue costs 2 points.

   + exec - 267 :
     analysis: Execution Time: 5 % - Vectorization Ratio: 100.00 % - Vector Length Use: 50.00 %
     Loop Computation Issues: 12
        [8] [SA] Presence of expensive FP instructions - Perform hoisting, change algorithm, use SVML or proper
            numerical library or perform value profiling (count the number of distinct input values). There are 2
            issues (= instructions) costing 4 points each.
        [4] [SA] Less than 10% of the FP ADD/SUB/MUL arithmetic operations are performed using FMA - Reorganize
            arithmetic expressions to exhibit potential for FMA. This issue costs 4 points.
     Data Access Issues: 0
        [0] [SA] Inefficient vectorization: more than 10% of the vector loads instructions are unaligned - When
            allocating arrays, don’t forget to align them. There are 0 issues ( = arrays) costing 2 points each

   + exec - 58  :
     analysis: Execution Time: 5 % - Vectorization Ratio: 100.00 % - Vector Length Use: 50.00 %
     Loop Computation Issues: 8
        [8] [SA] Presence of expensive FP instructions - Perform hoisting, change algorithm, use SVML or proper
            numerical library or perform value profiling (count the number of distinct input values). There are 2
            issues (= instructions) costing 4 points each.
     Data Access Issues: 0
        [0] [SA] Inefficient vectorization: more than 10% of the vector loads instructions are unaligned - When
            allocating arrays, don’t forget to align them. There are 0 issues ( = arrays) costing 2 points each

   + exec - 67  :
     analysis: Execution Time: 5 % - Vectorization Ratio: 100.00 % - Vector Length Use: 50.00 %
     Loop Computation Issues: 4
        [4] [SA] Presence of expensive FP instructions - Perform hoisting, change algorithm, use SVML or proper
            numerical library or perform value profiling (count the number of distinct input values). There are 1
            issues (= instructions) costing 4 points each.
     Data Access Issues: 0
        [0] [SA] Inefficient vectorization: more than 10% of the vector loads instructions are unaligned - When
            allocating arrays, don’t forget to align them. There are 0 issues ( = arrays) costing 2 points each

   + exec - 119 :
     analysis: Execution Time: 4 % - Vectorization Ratio: 100.00 % - Vector Length Use: 50.00 %
     Loop Computation Issues: 4
        [4] [SA] Presence of expensive FP instructions - Perform hoisting, change algorithm, use SVML or proper
            numerical library or perform value profiling (count the number of distinct input values). There are 1
            issues (= instructions) costing 4 points each.
     Data Access Issues: 0
        [0] [SA] Inefficient vectorization: more than 10% of the vector loads instructions are unaligned - When
            allocating arrays, don’t forget to align them. There are 0 issues ( = arrays) costing 2 points each



+====================================================================================================================+
+                                                 3  -  APPLICATION                                                  +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                               3.1  -  Categorization                                               +
+--------------------------------------------------------------------------------------------------------------------+

   Category | IO     | Exe    | System  | Others  | Memory | String | MPI   | TBB   | OMP   | Pthread | Math  |
  ----------+--------+--------+---------+---------+--------+--------+-------+-------+-------+---------+-------+
   Time (%) | 0.00   | 94.64  | 0.33    | 0.13    | 0.12   | 0.01   | 0.25  | 0.00  | 4.52  | 0.00    | 0.00  |




+--------------------------------------------------------------------------------------------------------------------+
+                                          3.2  -  Function Based Profiling                                          +
+--------------------------------------------------------------------------------------------------------------------+

   Buckets                    | Nb Functions              | Coverage                  | Cumulated Coverage        |
  ----------------------------+---------------------------+---------------------------+---------------------------+
   > 8%                       | 3                         | 66.11                     | 66.11                     |
   4% to 8%                   | 5                         | 24.58                     | 90.69                     |
   2% to 4%                   | 3                         | 7.90                      | 98.60                     |
   1% to 2%                   | 0                         | 0.00                      | 98.60                     |
   0.5% to 1%                 | 0                         | 0.00                      | 98.60                     |
   0.25% to 0.5%              | 2                         | 0.59                      | 99.19                     |
   0.125% to 0.25%            | 1                         | 0.17                      | 99.36                     |
   < 0.125%                   | 104                       | 0.64                      | 100.00                    |




+--------------------------------------------------------------------------------------------------------------------+
+                                            3.3  -  Loop Based Profiling                                            +
+--------------------------------------------------------------------------------------------------------------------+

   Buckets                    | Nb Loops                  | Coverage                  | Cumulated Coverage        |
  ----------------------------+---------------------------+---------------------------+---------------------------+
   > 8%                       | 0                         | 0.00                      | 0.00                      |
   4% to 8%                   | 7                         | 35.18                     | 35.18                     |
   2% to 4%                   | 17                        | 51.71                     | 86.89                     |
   1% to 2%                   | 4                         | 5.49                      | 92.37                     |
   0.5% to 1%                 | 2                         | 1.80                      | 94.17                     |
   0.25% to 0.5%              | 1                         | 0.28                      | 94.45                     |
   0.125% to 0.25%            | 0                         | 0.00                      | 94.45                     |
   < 0.125%                   | 75                        | 0.13                      | 94.59                     |


+====================================================================================================================+
+                                                  4  -  FUNCTIONS                                                   +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                              4.1  -  Top 10 Functions                                              +
+--------------------------------------------------------------------------------------------------------------------+

   Function                                               | Module              | Coverage (%)   | Time (s)       |
  --------------------------------------------------------+---------------------+----------------+----------------+
   advec_mom_kernel_mod_mp_advec_mom_kernel_.DIR.OMP.P... | exec                | 34.79          | 13.24          |
   advec_cell_kernel_module_mp_advec_cell_kernel_.DIR.... | exec                | 19.57          | 7.45           |
   pdv_kernel_module_mp_pdv_kernel_.DIR.OMP.PARALLEL.2    | exec                | 11.75          | 4.47           |
   ideal_gas_kernel_module_mp_ideal_gas_kernel_.DIR.OM... | exec                | 5.53           | 2.10           |
   accelerate_kernel_module_mp_accelerate_kernel_.DIR.... | exec                | 5.33           | 2.03           |
   reset_field_kernel_module_mp_reset_field_kernel_.DI... | exec                | 5.20           | 1.98           |
   kmp_flag_64<false, true>::wait(kmp_info*, int, void*)  | libiomp5.so         | 4.39           | 1.67           |
   flux_calc_kernel_module_mp_flux_calc_kernel_.DIR.OM... | exec                | 4.15           | 1.58           |
   calc_dt_kernel_module_mp_calc_dt_kernel_.DIR.OMP.PA... | exec                | 3.19           | 1.21           |
   revert_kernel_module_mp_revert_kernel_.DIR.OMP.PARA... | exec                | 2.54           | 0.97           |


+====================================================================================================================+
+                                                    5  -  LOOPS                                                     +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                                5.1  -  Top 10 Loops                                                +
+--------------------------------------------------------------------------------------------------------------------+

   Loop Id        | Module              | Source Location                                        | Coverage (%)   |
  ----------------+---------------------+--------------------------------------------------------+----------------+
   61             | exec                | PdV_kernel.f90:111-123,PdV_kernel.f90:131-135          | 6.24           |
   267            | exec                | ideal_gas_kernel.f90:49-55                             | 5.53           |
   58             | exec                | PdV_kernel.f90:75-87,PdV_kernel.f90:95-99              | 5.51           |
   67             | exec                | accelerate_kernel.f90:62-76                            | 5.32           |
   119            | exec                | advec_mom_kernel.f90:183-184                           | 4.29           |
   129            | exec                | advec_mom_kernel.f90:247-248                           | 4.14           |
   213            | exec                | flux_calc_kernel.f90:56-60                             | 4.14           |
   131            | exec                | advec_mom_kernel.f90:215-215,advec_mom_kernel.f90:2... | 3.92           |
   122            | exec                | advec_mom_kernel.f90:151-152,advec_mom_kernel.f90:1... | 3.66           |
   81             | exec                | advec_cell_kernel.f90:255-261                          | 3.60           |





+====================================================================================================================+
+                                                     6  -  CQA                                                      +
+====================================================================================================================+


+--------------------------------------------------------------------------------------------------------------------+
+                                                   6.1  -  Loops                                                    +
+--------------------------------------------------------------------------------------------------------------------+





      6.1.1  -  Loop 61 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/PdV_kernel.f90:111-123,131-135.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.1.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

23% of peak computational performance is used (7.50 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.1.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.1.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 16.00 to 12.50 cycles (1.28x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.1.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.1.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 16.00 to 13.50 cycles (1.19x speedup).


      6.1.1.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 12 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.1.1.6  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 9 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 9 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.1.1.7  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

27 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.1.1.8  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 120 FP arithmetical operations:
 - 72: addition or subtraction (12 inside FMA instructions)
 - 40: multiply (12 inside FMA instructions)
 - 8: divide
The binary loop is loading 856 bytes (107 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.1.1.9  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.13 FP operations per loaded or stored byte.







      6.1.2  -  Loop 267 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/ideal_gas_kernel.f90:49-55.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.2.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

5% of peak computational performance is used (1.88 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.2.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.2.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 17.00 to 4.00 cycles (4.25x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.2.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.2.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 17.00 to 4.00 cycles (4.25x speedup).




      6.1.2.1.5  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 3 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 3 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.2.1.6  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

8 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.2.1.7  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 32 FP arithmetical operations:
 - 24: multiply
 - 4: divide
 - 4: square root
The binary loop is loading 64 bytes (8 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.2.1.8  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.25 FP operations per loaded or stored byte.







      6.1.3  -  Loop 58 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/PdV_kernel.f90:75-87,95-99.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.3.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

17% of peak computational performance is used (5.50 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.3.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.3.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 16.00 to 8.50 cycles (1.88x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.3.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.3.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 16.00 to 9.50 cycles (1.68x speedup).


      6.1.3.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 12 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.3.1.6  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 9 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 9 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.3.1.7  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

19 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.3.1.8  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 88 FP arithmetical operations:
 - 40: addition or subtraction (12 inside FMA instructions)
 - 40: multiply (12 inside FMA instructions)
 - 8: divide
The binary loop is loading 568 bytes (71 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.3.1.9  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.14 FP operations per loaded or stored byte.







      6.1.4  -  Loop 67 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/accelerate_kernel.f90:62-76.

The related source loop is not unrolled or unrolled with no peel/tail loop.

      6.1.4.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

38% of peak computational performance is used (12.17 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.4.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).


Workaround
Read the "512-bits vectorization" report at "Potential" confidence level.


      6.1.4.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of FP multiply or FMA (fused multiply-add) operations (the FP multiply/FMA unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 11.50 to 8.83 cycles (1.30x speedup).


Workaround
Reduce the number of FP multiply/FMA instructions




      6.1.4.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.4.1.4  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 44 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
Estimated speedup by perfect pairing: 1.21x.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.4.1.5  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 18 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 18 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.4.1.6  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

24 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.4.1.7  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 140 FP arithmetical operations:
 - 76: addition or subtraction (44 inside FMA instructions)
 - 60: multiply (44 inside FMA instructions)
 - 4: divide
The binary loop is loading 736 bytes (92 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.4.1.8  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.17 FP operations per loaded or stored byte.


      6.1.4.1.9  -  Unroll opportunity
  ----------------------------------------------------------------------------------------------------------

Loop is potentially data access bound.

Workaround
Unroll your loop if trip count is significantly higher than target unroll factor and if some data references are common to consecutive iterations. This can be done manually. Or with the unroll (resp. unroll_and_jam) directive on top of the inner (resp. surrounding) loop. You can enforce an unroll factor: !DIR$ UNROLL_AND_JAM(N), UNROLL_AND_JAM=N, UNROLL(N) or UNROLL=N







      6.1.5  -  Loop 119 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/advec_mom_kernel.f90:183-184.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.5.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

6% of peak computational performance is used (2.00 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.5.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.5.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 8.00 to 1.83 cycles (4.36x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.5.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.5.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 8.00 to 1.67 cycles (4.80x speedup).


      6.1.5.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 4 FMA (fused multiply-add) operations.




      6.1.5.1.6  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 3 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 3 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.5.1.7  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

3 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.5.1.8  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 16 FP arithmetical operations:
 - 8: addition or subtraction (4 inside FMA instructions)
 - 4: multiply (all inside FMA instructions)
 - 4: divide
The binary loop is loading 160 bytes (20 double precision FP elements).
The binary loop is storing 32 bytes (4 double precision FP elements).


      6.1.5.1.9  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.08 FP operations per loaded or stored byte.







      6.1.6  -  Loop 129 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/advec_mom_kernel.f90:247-248.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.6.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

6% of peak computational performance is used (2.00 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.6.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.6.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 8.00 to 1.83 cycles (4.36x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.6.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.6.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 8.00 to 1.67 cycles (4.80x speedup).


      6.1.6.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 4 FMA (fused multiply-add) operations.




      6.1.6.1.6  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 3 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 3 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.6.1.7  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

3 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.6.1.8  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 16 FP arithmetical operations:
 - 8: addition or subtraction (4 inside FMA instructions)
 - 4: multiply (all inside FMA instructions)
 - 4: divide
The binary loop is loading 160 bytes (20 double precision FP elements).
The binary loop is storing 32 bytes (4 double precision FP elements).


      6.1.6.1.9  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.08 FP operations per loaded or stored byte.







      6.1.7  -  Loop 213 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/flux_calc_kernel.f90:56-60.

The related source loop is not unrolled or unrolled with no peel/tail loop.

      6.1.7.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

25% of peak computational performance is used (8.00 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.7.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).


Workaround
Read the "512-bits vectorization" report at "Potential" confidence level.


      6.1.7.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of FP multiply or FMA (fused multiply-add) operations (the FP multiply/FMA unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 5.00 to 4.00 cycles (1.25x speedup).


Workaround
Reduce the number of FP multiply/FMA instructions




      6.1.7.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.7.1.4  -  FMA
  ----------------------------------------------------------------------------------------------------------

Presence of both ADD/SUB and MUL operations.

Workaround
 - Pass to your compiler a micro-architecture specialization option:
  * use axHost or xHost
 - Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
Estimated speedup by perfect pairing: 1.50x.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).





      6.1.7.1.5  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 4 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 4 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.7.1.6  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

10 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.7.1.7  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 40 FP arithmetical operations:
 - 24: addition or subtraction
 - 16: multiply
The binary loop is loading 320 bytes (40 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.7.1.8  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.10 FP operations per loaded or stored byte.


      6.1.7.1.9  -  Unroll opportunity
  ----------------------------------------------------------------------------------------------------------

Loop is potentially data access bound.

Workaround
Unroll your loop if trip count is significantly higher than target unroll factor and if some data references are common to consecutive iterations. This can be done manually. Or with the unroll (resp. unroll_and_jam) directive on top of the inner (resp. surrounding) loop. You can enforce an unroll factor: !DIR$ UNROLL_AND_JAM(N), UNROLL_AND_JAM=N, UNROLL(N) or UNROLL=N







      6.1.8  -  Loop 131 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/advec_mom_kernel.f90:215,227-241.

The related source loop is not unrolled or unrolled with no peel/tail loop.
The structure of this loop is probably <if then [else] end>.

The presence of multiple execution paths is typically the main/first bottleneck.
Try to simplify control inside loop: ideally, try to remove all conditional expressions, for example by (if applicable):
 - hoisting them (moving them outside the loop)
 - turning them into conditional moves, MIN or MAX


Ex: if (x<0) x=0 => x = max(0,x) (Fortran instrinsic procedure)


      6.1.8.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

Warnings:
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K3] is unknown
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K2] is unknown

7% of peak computational performance is used (2.46 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.8.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is highly vectorized.
Only 40% of vector register length is used (average across all SSE/AVX instructions).


Details
91% of SSE/AVX instructions are used in vector version (process two or more data elements in vector registers):
 - 85% of SSE/AVX loads are used in vector version.
 - 88% of SSE/AVX multiply instructions are used in vector version.
 - 66% of SSE/AVX divide and square root instructions are used in vector version.
 - 91% of SSE/AVX instructions that are not load, store, addition, subtraction nor multiply instructions are used in vector version.


Workaround
Read the "512-bits vectorization" report at "Potential" confidence level.


      6.1.8.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of FP multiply or FMA (fused multiply-add) operations (the FP multiply/FMA unit is a bottleneck).


Workaround
Reduce the number of FP multiply/FMA instructions




      6.1.8.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.8.1.4  -  Masked instructions
  ----------------------------------------------------------------------------------------------------------

Detected masked instructions.

Details
Vector registers are partially exploited, which is expected if your loop is irregular or mixes elements of different sizes.

Workaround
If your loop is irregular, try to remove or hoist conditional structures out of your loop. If it mixes elements of different sizes, try to uniformize them.


      6.1.8.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 12 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
Estimated speedup by perfect pairing: 1.05x.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.8.1.6  -  Complex instructions
  ----------------------------------------------------------------------------------------------------------

Detected COMPLEX INSTRUCTIONS.


Details
These instructions generate more than one micro-operation and only one of them can be decoded during a cycle and the extra micro-operations increase pressure on execution units.
 - VGATHERQPD: 5 occurrences<<list_path_1_complex_1>>
 - VPMULLQ: 4 occurrences<<list_path_1_complex_2>>



      6.1.8.1.7  -  Slow data structures access
  ----------------------------------------------------------------------------------------------------------

Detected data structures (typically arrays) that cannot be efficiently read/written

Details
 - Irregular (variable stride) or indirect: 4 occurrence(s)
Non-unit stride (uncontiguous) accesses are not efficiently using data caches


Workaround
Try to remove indirect accesses. If applicable, precompute elements out of the innermost loop.


      6.1.8.1.8  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 2 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 2 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.8.1.9  -  Gather/scatter instructions
  ----------------------------------------------------------------------------------------------------------

Detected gather/scatter instructions (typically caused by indirect accesses). By removing them, you can lower the cost of an iteration from 26.83 to 21.67 cycles (1.24x speedup).

Details
 - VGATHERQPD: 5 occurrences<<list_path_1_gather_scatter_1>>


Workaround
Try to simplify your code and/or replace indirect accesses with unit-stride ones.


      6.1.8.1.10  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

2 SSE or AVX instructions are processing arithmetic or math operations on double precision FP elements in scalar mode (one at a time).
18 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.8.1.11  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 66 FP arithmetical operations:
 - 28: addition or subtraction (12 inside FMA instructions)
 - 29: multiply (12 inside FMA instructions)
 - 9: divide
The binary loop is loading 200 bytes (25 double precision FP elements).
The binary loop is storing 32 bytes (4 double precision FP elements).


      6.1.8.1.12  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.28 FP operations per loaded or stored byte.




      6.1.8.2  -  Path 2
  ----------------------------------------------------------------------------------------------------------

Warnings:
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K3] is unknown
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K2] is unknown

7% of peak computational performance is used (2.46 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.8.2.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is highly vectorized.
Only 41% of vector register length is used (average across all SSE/AVX instructions).


Details
92% of SSE/AVX instructions are used in vector version (process two or more data elements in vector registers):
 - 88% of SSE/AVX multiply instructions are used in vector version.
 - 66% of SSE/AVX divide and square root instructions are used in vector version.
 - 91% of SSE/AVX instructions that are not load, store, addition, subtraction nor multiply instructions are used in vector version.


Workaround
Read the "512-bits vectorization" report at "Potential" confidence level.


      6.1.8.2.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of FP multiply or FMA (fused multiply-add) operations (the FP multiply/FMA unit is a bottleneck).


Workaround
Reduce the number of FP multiply/FMA instructions




      6.1.8.2.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.8.2.4  -  Masked instructions
  ----------------------------------------------------------------------------------------------------------

Detected masked instructions.

Details
Vector registers are partially exploited, which is expected if your loop is irregular or mixes elements of different sizes.

Workaround
If your loop is irregular, try to remove or hoist conditional structures out of your loop. If it mixes elements of different sizes, try to uniformize them.


      6.1.8.2.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 12 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
Estimated speedup by perfect pairing: 1.05x.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.8.2.6  -  Complex instructions
  ----------------------------------------------------------------------------------------------------------

Detected COMPLEX INSTRUCTIONS.


Details
These instructions generate more than one micro-operation and only one of them can be decoded during a cycle and the extra micro-operations increase pressure on execution units.
 - VGATHERQPD: 5 occurrences<<list_path_2_complex_1>>
 - VPMULLQ: 4 occurrences<<list_path_2_complex_2>>



      6.1.8.2.7  -  Slow data structures access
  ----------------------------------------------------------------------------------------------------------

Detected data structures (typically arrays) that cannot be efficiently read/written

Details
 - Irregular (variable stride) or indirect: 5 occurrence(s)
Non-unit stride (uncontiguous) accesses are not efficiently using data caches


Workaround
Try to remove indirect accesses. If applicable, precompute elements out of the innermost loop.


      6.1.8.2.8  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 2 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 2 occurrences<<list_path_2_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.8.2.9  -  Gather/scatter instructions
  ----------------------------------------------------------------------------------------------------------

Detected gather/scatter instructions (typically caused by indirect accesses). By removing them, you can lower the cost of an iteration from 26.83 to 21.67 cycles (1.24x speedup).

Details
 - VGATHERQPD: 5 occurrences<<list_path_2_gather_scatter_1>>


Workaround
Try to simplify your code and/or replace indirect accesses with unit-stride ones.


      6.1.8.2.10  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

2 SSE or AVX instructions are processing arithmetic or math operations on double precision FP elements in scalar mode (one at a time).
18 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.8.2.11  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 66 FP arithmetical operations:
 - 28: addition or subtraction (12 inside FMA instructions)
 - 29: multiply (12 inside FMA instructions)
 - 9: divide
The binary loop is loading 192 bytes (24 double precision FP elements).
The binary loop is storing 32 bytes (4 double precision FP elements).


      6.1.8.2.12  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.29 FP operations per loaded or stored byte.







      6.1.9  -  Loop 122 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/advec_mom_kernel.f90:151-152,158-177.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.9.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

Warnings:
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K2] is unknown
 - The number of fused uops of the instruction [VPCMPEQB	%XMM0,%XMM0,%K1] is unknown

8% of peak computational performance is used (2.83 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.9.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is highly vectorized.
Only 40% of vector register length is used (average across all SSE/AVX instructions).


Details
98% of SSE/AVX instructions are used in vector version (process two or more data elements in vector registers):
 - 96% of SSE/AVX instructions that are not load, store, addition, subtraction nor multiply instructions are used in vector version.



      6.1.9.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 24.00 to 18.83 cycles (1.27x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.9.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.9.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 24.00 to 18.67 cycles (1.29x speedup).


      6.1.9.1.5  -  Masked instructions
  ----------------------------------------------------------------------------------------------------------

Detected masked instructions.

Details
Vector registers are partially exploited, which is expected if your loop is irregular or mixes elements of different sizes.

Workaround
If your loop is irregular, try to remove or hoist conditional structures out of your loop. If it mixes elements of different sizes, try to uniformize them.


      6.1.9.1.6  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 8 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.9.1.7  -  Complex instructions
  ----------------------------------------------------------------------------------------------------------

Detected COMPLEX INSTRUCTIONS.


Details
These instructions generate more than one micro-operation and only one of them can be decoded during a cycle and the extra micro-operations increase pressure on execution units.
 - VGATHERQPD: 5 occurrences<<list_path_1_complex_1>>



      6.1.9.1.8  -  Slow data structures access
  ----------------------------------------------------------------------------------------------------------

Detected data structures (typically arrays) that cannot be efficiently read/written

Details
 - Irregular (variable stride) or indirect: 3 occurrence(s)
Non-unit stride (uncontiguous) accesses are not efficiently using data caches


Workaround
Try to remove indirect accesses. If applicable, precompute elements out of the innermost loop.


      6.1.9.1.9  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 3 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 3 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.9.1.10  -  Gather/scatter instructions
  ----------------------------------------------------------------------------------------------------------

Detected gather/scatter instructions (typically caused by indirect accesses).

Details
 - VGATHERQPD: 5 occurrences<<list_path_1_gather_scatter_1>>


Workaround
Try to simplify your code and/or replace indirect accesses with unit-stride ones.


      6.1.9.1.11  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

20 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.9.1.12  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 68 FP arithmetical operations:
 - 28: addition or subtraction (8 inside FMA instructions)
 - 28: multiply (8 inside FMA instructions)
 - 12: divide
The binary loop is loading 224 bytes (28 double precision FP elements).
The binary loop is storing 32 bytes (4 double precision FP elements).


      6.1.9.1.13  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.27 FP operations per loaded or stored byte.







      6.1.10  -  Loop 81 from exec
  ==========================================================================================================

The loop is defined in /beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/build/CloverLeaf1.3-FC/CloverLeaf_ref/kernels/advec_cell_kernel.f90:255-261.

It is main loop of related source loop which is unrolled by 4 (including vectorization).

      6.1.10.1  -  Path 1
  ----------------------------------------------------------------------------------------------------------

7% of peak computational performance is used (2.50 out of 32.00 FLOP per cycle (GFLOPS @ 1GHz))

      6.1.10.1.1  -  Vectorization
  ----------------------------------------------------------------------------------------------------------

Your loop is vectorized, but using only 256 out of 512 bits (AVX/AVX2 instructions on AVX-512 processors).
<<image_4x64_512>>

Details
All SSE/AVX instructions are used in vector version (process two or more data elements in vector registers).



      6.1.10.1.2  -  Execution units bottlenecks
  ----------------------------------------------------------------------------------------------------------

Performance is limited by execution of divide and square root operations (the divide/square root unit is a bottleneck).

By removing all these bottlenecks, you can lower the cost of an iteration from 16.00 to 3.67 cycles (4.36x speedup).


Workaround
 - Reduce the number of division or square root instructions:
  * If denominator is constant over iterations, use reciprocal (replace x/y with x*(1/y)). Check precision impact. This will be done by your compiler with no-prec-div or Ofast
 - Check whether you really need double precision. If not, switch to single precision to speedup execution





      6.1.10.1.3  -  512-bits vectorization
  ----------------------------------------------------------------------------------------------------------

On some x86 processors supporting 512-bits vectorization, compilers are often too conservative and limit vectorization to 256 bits. Performance can then be improved by enforcing 512-bits vectorization, especially with many vectorized and high trip count loops. 512-bits vectorization performance overhead (compared to 256-bits) is generally lower on newer processors.


Workaround
Recompile with -qopt-zmm-usage=high


      6.1.10.1.4  -  Expensive FP math instructions/calls
  ----------------------------------------------------------------------------------------------------------

Detected performance impact from expensive FP math instructions/calls.
By removing/reexpressing them, you can lower the cost of an iteration from 16.00 to 4.50 cycles (3.56x speedup).


      6.1.10.1.5  -  FMA
  ----------------------------------------------------------------------------------------------------------

Detected 4 FMA (fused multiply-add) operations.
Presence of both ADD/SUB and MUL operations.

Workaround
Try to change order in which elements are evaluated (using parentheses) in arithmetic expressions containing both ADD/SUB and MUL operations to enable your compiler to generate FMA instructions wherever possible.
For instance a + b*c is a valid FMA (MUL then ADD).
However (a+b)* c cannot be translated into an FMA (ADD then MUL).




      6.1.10.1.6  -  Vector unaligned load/store instructions
  ----------------------------------------------------------------------------------------------------------

Detected 4 optimal vector unaligned load/store instructions.


Details
 - VMOVUPD: 4 occurrences<<list_path_1_vec_align_1>>


Workaround
Use vector aligned instructions:
 1) align your arrays on 64 bytes boundaries: compile with -align array64byte (remark: not affects arrays in COMMON blocks).
 2) inform your compiler that your arrays are vector aligned: Append !DIR$ VECTOR ALIGNED to the loop if all accessed arrays are aligned, or !DIR$ ASSUME_ALIGNED FOO: 64 if only FOO is aligned.
<<image_vec_align>>


      6.1.10.1.7  -  Type of elements and instruction set
  ----------------------------------------------------------------------------------------------------------

9 AVX instructions are processing arithmetic or math operations on double precision FP elements in vector mode (four at a time).



      6.1.10.1.8  -  Matching between your loop (in the source code) and the binary loop
  ----------------------------------------------------------------------------------------------------------

The binary loop is composed of 40 FP arithmetical operations:
 - 24: addition or subtraction (4 inside FMA instructions)
 - 8: multiply (4 inside FMA instructions)
 - 8: divide
The binary loop is loading 288 bytes (36 double precision FP elements).
The binary loop is storing 64 bytes (8 double precision FP elements).


      6.1.10.1.9  -  Arithmetic intensity
  ----------------------------------------------------------------------------------------------------------

Arithmetic intensity is 0.11 FP operations per loaded or stored byte.





[MAQAO] Info: STOP THE REPORT GENERATION
[MAQAO] Info: 
[MAQAO] Info: If your application produces files, they can be found in directory "/beegfs/hackathon/users/eoseret/qaas_runs_test/178-661-4073/intel/CloverLeaf1.3-FC/run/oneview_runs/defaults/orig/oneview_run_1786614283"
[MAQAO] Info: 
